Communication terminal, communication method and computer readable information recording medium

ABSTRACT

A communication terminal receives display data for displaying certain data on a screen of a display device and image data of an optical-to-electrical-converted image from another communication terminal via a communication network, includes a storage part storing reception information indicating whether the display data has been received by the communication terminal and arrangement information indicating at least one of a size and a position of the image based on the image data, in a manner of associating with one another; a reception part receiving the display data and the image data; an extraction part extracting, based on the reception information, the corresponding arrangement information from the storage part; and a display control part displaying the image using the image data based on the arrangement information extracted by the extraction part.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application is a continuation of and claims thebenefit of priority under 35 U.S.C. §120 from U.S. application Ser. No.13/883,481, filed May 3, 2013, which is a national stage ofInternational Application No. PCT/JP2011/075522, filed Oct. 28, 2011,which is based on and claims the benefit of priority under 35 U.S.C.§119 from Japanese Priority Patent Application No. 2010-247551, filedNov. 4, 2010, Japanese Priority Patent Application No. 2010-267757,filed Nov. 30, 2010 and Japanese Priority Patent Application No.2011-168464, filed Aug. 1, 2011, the entire contents of all of which arehereby incorporated by reference.

TECHNICAL FIELD

The disclosure relates to a communication terminal (or remotecommunication terminal) which is capable of receiving display data fordisplaying certain data on a screen of a display device and image dataof an optical-to-electrical-converted image, from another communicationterminal (or remote communication terminal) via a communication network(or remote communication network).

BACKGROUND ART

Recently, along with a demand to reduce business trip costs and businesstrip man hours, communication systems (or remote communication systems)carrying out a TV conference using a communication network (or remotecommunication network) such as the Internet have become widespread. Insuch a communication system, display data for displaying material dataon a screen, image data and/or voice data are transmitted among pluralcommunication terminals (or remote communication terminals), and thus aTV conference is achieved. Further, thanks to recent improvements ofbroadband environments, it becomes possible to transmit image data ofhigh image quality and voice data of high voice quality. Therefore, itbecomes possible to easily determine a situation of another person withwhom a TV conference is being conducted, and it becomes possible toimprove fullness of mutual understanding achieved by conversation duringthe TV conference.

Further, since a communication terminal (or remote communicationterminal) becomes capable of transmitting and receiving a large amountof display data simultaneously, it becomes possible for each participantof a TV conference to read conference material while determining asituation of another participant, as a result of image data and thedisplay data being displayed from a display device simultaneously (seeU.S. Pat. No. 6,760,749).

However, there may be a case where no material is used in a TVconference. In such a case, nothing is displayed at an area prepared fordisplaying material data on a display device, and an area prepared fordisplaying image data of a participant of the TV conference is as is inthe original state. It is possible to omit the area displaying nothingso that the area displaying image data is widened, and the image databecomes easier to watch. However, for this purpose, the areas on thedisplay device are to be changed by manual operations.

Further, after the area displaying the image data is widened, there maybe a case where during the TV conference, it becomes necessary to usematerial. Also in such a case, the areas on the display device are to bechanged to widen the area displaying the material so that the materialbecomes easy to be seen, and thus, manual operations become necessaryagain. Such manual operations may be troublesome for the participantswho are conducting the TV conference.

SUMMARY OF INVENTION

According to an embodiment of the present invention, a communicationterminal (or remote communication terminal), which is capable ofreceiving display data for displaying certain data on a display deviceand image data of an optical-to-electrical-converted image, from anothercommunication terminal (or remote communication terminal) via acommunication network (or remote communication network), includes astorage part configured to store reception information indicatingwhether the display data has been received and arrangement informationindicating at least one of a size and a position of an image based onthe image data to be displayed on the display device in a manner ofassociating them together; a reception part configured to receive thedisplay data and the image data; an extraction part configured toextract, based on the reception information, the correspondingarrangement information from the storage part; and a display controlpart configured to display, based on the arrangement informationextracted by the extraction part, images according to the image data onthe screen of the display device.

Other objects, features and advantages of the present invention willbecome more apparent from the following detailed description when readin conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a general configuration of an example of a communicationsystem according to a first embodiment of the present invention;

FIG. 2 shows an external appearance (perspective view) of an example ofa communication terminal according to the first embodiment;

FIG. 3 shows a hardware configuration of an example of the communicationterminal according to the first embodiment;

FIG. 4 shows a hardware configuration of an example of each of acommunication management system, a relay apparatus and an external inputapparatus according to the first embodiment;

FIGS. 5A and 5B show a block diagram of an example of each of theterminals, the apparatuses and the system included in the communicationsystem according to the first embodiment;

FIG. 6A shows an example of image data of a low resolution forillustrating image quality according to the first and secondembodiments;

FIG. 6B shows an example of image data of a medium resolution forillustrating image quality according to the first and secondembodiments;

FIG. 6C shows an example of image data of a high resolution forillustrating image quality according to the first and secondembodiments;

FIG. 7 shows a concept of an example of a change quality managementtable according to the first and second embodiments;

FIG. 8 shows a concept of an example of a relay apparatus managementtable according to the first and second embodiments;

FIG. 9 shows a concept of an example of a terminal authenticationmanagement table according to the first and second embodiments;

FIG. 10 shows a concept of an example of a terminal management tableaccording to the first and second embodiments;

FIG. 11 shows a concept of an example of a destination list managementtable according to the first and second embodiments;

FIG. 12 shows a concept of an example of a session management tableaccording to the first and second embodiments;

FIG. 13 shows a concept of an example of an address priority managementtable according to the first and second embodiments;

FIG. 14 shows a concept of an example of a transmission rate prioritymanagement table according to the first and second embodiments;

FIG. 15 shows a concept of an example of a quality management tableaccording to the first and second embodiments;

FIG. 16 shows an example of a sequence diagram of a process of managingstate information indicating an operating state of each of the relayapparatuses according to the first embodiment;

FIG. 17 shows an example of a sequence diagram of a process of apreparation step for starting communication between the communicationterminals according to the first embodiment;

FIG. 18 shows an example of a sequence diagram of a process of narrowingdown the relay apparatuses according to the first embodiment;

FIG. 19 shows an example of a flowchart of a process of narrowing downthe relay apparatuses according to the first embodiment;

FIG. 20 shows a calculation state of point counts of priority whennarrowing down the relay apparatuses according to the first and secondembodiments;

FIG. 21 shows an example of a sequence diagram of a process of selectingthe relay apparatus by the communication terminal according to the firstembodiment;

FIG. 22 shows an example of a flowchart of a process of selecting therelay apparatus by the communication terminal according to the firstembodiment;

FIG. 23 shows an example of a sequence diagram of a process oftransmitting image data and voice data between the communicationterminals according to the first embodiment;

FIG. 24 shows a concept of an arrangement information management tablestored by a storage part according to the first embodiment;

FIG. 25 shows a concept of an area management table stored by thestorage part according to the first embodiment;

FIGS. 26A and 26B show examples of screen pages displayed based onarrangement information according to the first embodiment;

FIG. 27 shows a functional block diagram of an external input apparatusaccording to the first embodiment;

FIG. 28 shows a concept of a material management table according to thefirst and second embodiments;

FIG. 29 is a sequence diagram showing a process of displaying image dataand display data by the communication terminal according to the firstembodiment;

FIG. 30 is a flowchart showing a process of determining, by thecommunication terminal, arrangement information used for displayingimage data and display data according to the first embodiment;

FIG. 31 is a sequence diagram showing a process of stopping transmissionof display data according to the first embodiment;

FIG. 32 is a flowchart showing a process of determining arrangementinformation based on operations carried out by a user according to thefirst embodiment;

FIG. 33 is a state transition diagram showing a state transition ofarrangement information based on operations carried out by a useraccording to the first embodiment;

FIG. 34 shows a general configuration of an example of a remotecommunication system according to the second embodiment of the presentinvention;

FIG. 35 shows a hardware configuration of an example of the remotecommunication terminal according to the second embodiment;

FIG. 36 shows a hardware configuration of an example of each of a remotecommunication management system, a relay apparatus and an external inputapparatus according to the second embodiment;

FIGS. 37A and 37B show a block diagram of an example of the terminals,the apparatuses and the system included in the remote communicationsystem according to the second embodiment;

FIG. 38 shows an example of a sequence diagram of a process of managingstate information indicating an operating state of each of the relayapparatuses according to the second embodiment;

FIG. 39 shows an example of a sequence diagram of a process of apreparation step for starting remote communication between the remotecommunication terminals according to the second embodiment;

FIG. 40 shows an example of a sequence diagram of a process of narrowingdown the relay apparatuses according to the second embodiment;

FIG. 41 shows an example of a flowchart of a process of narrowing downthe relay apparatuses according to the second embodiment;

FIG. 42 shows an example of a sequence diagram of a process of selectingthe relay apparatus by the remote communication terminal according tothe second embodiment;

FIG. 43 shows an example of a flowchart of a process of selecting therelay apparatus by the remote communication terminal according to thesecond embodiment;

FIG. 44 shows an example of a sequence diagram of a process oftransmitting image data and voice data between the remote communicationterminals according to the second embodiment;

FIG. 45 shows a concept of an arrangement information management tablestored by a storage part according to the second embodiment;

FIGS. 46A and 46B show examples of screen pages in which material dataand image data are displayed based on an arrangement drawing asarrangement information stored in the storage part according to thesecond embodiment;

FIG. 47 shows a functional block diagram of an external input apparatusaccording to the second embodiment;

FIG. 48 is a sequence diagram showing a process of displaying image dataand display data by the remote communication terminal according to thesecond embodiment;

FIG. 49 is a flowchart showing a process of determining, by the remotecommunication terminal, the arrangement in the process of displayingimage data and display data according to the second embodiment;

FIGS. 50A, 50B show examples of images displayed on the remotecommunication terminal; and

FIG. 51 shows an example of an image displayed on the remotecommunication terminal.

DESCRIPTION OF EMBODIMENTS

According to embodiments of the present invention, using arrangementinformation extracted from a storage part based on whether acommunication terminal (or remote communication terminal) receivesdisplay data displayed on an external input apparatus of anotherparticipant of a TV conference, a screen of a display device isefficiently used, and thereby, it is possible to avoid troublesomenesswhich may otherwise occur for the participants of the TV conference.

First Embodiment

With reference to FIGS. 1 through 33, a first embodiment of the presentinvention will be described.

Overall Configuration of First Embodiment

FIG. 1 is a general configuration of a communication system 1 used for aTV conference according to the first embodiment. Using FIG. 1, a generalconfiguration of the first embodiment will now be described. It is notedthat the term “TV conference” means also one generally called “videoconference”.

First, the communication system 1 shown in FIG. 1 includes pluralcommunication terminals (10 aa, 10 ab, . . . , 10 db), display devices(100 aa, 100 ab, . . . , 100 db) for the respective communicationterminals (10 aa, 10 ab, . . . , 10 db), plural relay apparatuses (30 a,30 b, 30 c and 30 d), a communication management system 50, anarrangement information providing system 80 and a program providingsystem 90.

It is noted that in a case of indicating any communication terminal ofthe plural communication terminals (10 aa, 10 ab, . . . , 10 db),“communication terminal 10” is used. In a case of indicting any displaydevice of the plural display devices (100 aa, 100 ab, . . . , 100 db),“display device 100” is used. In a case of indicating any one of theplural relay apparatuses (30 a, 30 b, 30 c and 30 d), “relay apparatus30” is used.

The communication terminal 10 transmits and receives image data, voicedata and so forth to and from another communication terminal 10.Although a case will be described where images of image data (obtainedby a camera 112, for example) is video (moving images), images of imagedata may be static images. Further, images of image data may includeboth video (moving images) and static images. The relay apparatus 30relays image data and voice data between plural communication terminals10. The communication management system 50 manages, in a unifyingmanner, the communication terminals 10 and the relay apparatuses 30. Thearrangement information providing system 80 provides, to thecommunication terminal 10, arrangement information used for arrangingimage data and material data (which will be described later) on thescreen of the communication terminal 10.

The external input apparatus 40 is connected with the communicationterminal 10, and transmits display data, obtained when obtaining ascreen page that displays material data, to the communication terminal10. The material data means, for example, data produced using documentcomposition software, spreadsheet software, presentation software orsuch. “Obtaining a screen page” means storing a static image or ananimation displayed on a display device of a computer.

Plural routers (70 a, 70 b, . . . , 70 g) shown in FIG. 1 select optimumpaths for the image data and voice data. It is noted that hereinafter,in a case of indicating any one of the plural routers (70 a, 70 b, . . ., 70 g), a “router 70” is used. A program providing system 90 includes aHD (Hard Disk) (not shown), stores a program for each of thecommunication terminals 10 to carry out various functions or causingeach of the communication terminals 10 to function as various functionalparts, and is capable of transmitting the program to each of thecommunication terminals 10. Further, the HD of the program providingsystem 90 also stores a program for each of the relay apparatuses 30 tocarry out various functions or causing each of the relay apparatuses 30to function as various functional parts, and capable of transmitting theprogram to each of the relay apparatuses 30. Further, the HD of theprogram providing system 90 also stores a program for communicationmanagement to carry out various functions or causing the communicationmanagement apparatus 50 to function as various functional parts, and iscapable of transmitting the program to the communication managementapparatus 50.

The communication terminals 10 aa, 10 ab, the relay apparatuses 30 a andthe router 70 a are connected together by a LAN 2 a in such a mannerthat they can carry out mutual communication. The communicationterminals 10 ba, 10 bb, the relay apparatus 30 b and the router 70 b areconnected together by a LAN 2 b in such a manner that they can carry outmutual communication. The LAN 2 a and the LAN 2 b are connected togetherby a private line 2 ab including the router 70 c in such a manner thatthey can carry out mutual communication, and are built in a certain areaA. For example, the area A is Japan, the LAN 2 a is built in any companyin Tokyo, and the LAN 2 b is built in any company in Osaka.

On the other hand, the communication terminals 10 ca, 10 cb, the relayapparatus 30 c and the router 70 d are connected together by a LAN 2 cin such a manner that they can carry out mutual communication. Thecommunication terminals 10 da, 10 db, the relay apparatuses 30 d and therouter 70 e are connected together by a LAN 2 d in such a manner thatthey can carry out mutual communication. The LAN 2 c and the LAN 2 d areconnected together by a private line 2 cd including the router 70 f insuch a manner that they can carry out mutual communication, and arebuilt in a certain area B. For example, the area B is the USA, the LAN 2c is built in any company in New York, and the LAN 2 d is built in anycompany in Washington, D.C. The area A and the area B are connectedtogether in such a manner that they can carry out mutual communicationby the Internet 2 i from the routers (70 c and 70 f), respectively.

Further, the communication management system 50, the arrangementinformation providing system 80 and the program providing system 90 areconnected with the communication terminals 10 and the relay apparatuses30 in such a manner that they can carry out mutual communication by theInternet 2 i. The communication management system 50, the arrangementinformation providing system 80 and the program providing system 90 maybe installed in the area A or the area B, or may be installed in anotherarea.

It is noted that in the first embodiment, the communication network 2includes the LAN 2 a, the LAN 2 b, the private line tab, the Internet 2i, the private line 2 cd, the LAN 2 c and the LAN 2 d.

Further, in FIG. 1, four numerals below each of the communicationterminals 10, the relay apparatuses 30, the communication managementsystem 50 and the routers 70 show an IP address according to the commonIPv4 in a simplified manner. For example, the IP address of thecommunication terminal 10 aa is “1.2.1.3”. IPv6 may be used instead ofIPv4. However, for the purpose of simplifying the description, thedescription will be made using IPv4.

Hardware Configuration of First Embodiment

Next, a hardware configuration of the first embodiment will bedescribed. A case will be described where in a situation where a delayoccurs in reception of image data by the communication terminal 10acting as a relay destination, the relay apparatus 30 changes theresolution of the image of the image data, and then, the image data istransmitted to the communication terminal 10 acting as the relaydestination.

FIG. 2 shows an external appearance of the communication terminal 10according to the first embodiment. Hereinafter, the longitudinaldirection of the communication terminal 10 will be referred to as anX-axis direction, the direction perpendicular to the X-axis direction ona horizontal plane will be referred to as a Y-axis direction, and thedirection (vertical direction) perpendicular to the X-axis direction andthe Y-axis direction will be referred to as a Z-axis direction.

As shown in FIG. 2, the communication terminal 10 includes a housing1100, an arm 1200 and a camera housing 1300. On a rear side wall 1110 ofthe housing 1100, an air suction surface (not shown) including pluralair suction holes is provided. On a front side wall 1120 of the housing1100, an air discharge surface 1121 including plural air discharge holesis provided. Thereby, as a result of a cooling fan (not shown) providedin the inside of the housing 1100 being driven, air behind the terminal10 is taken in via the air suction surface, and the air is discharged tothe front side of the communication terminal 10 via the air dischargesurface 1121. On a right side wall 1130 of the housing 1100, a soundcollecting hole 1131 is formed, and a voice, a sound, a noise or such iscollected by means of a microphone 114 (described later) provided in theinside of the housing 1100.

On a top surface of the housing 1100 on an area near the right side wall1130, an operations panel 1150 is provided. On the operations panel1150, plural operating buttons (108 a through 108 e) described later, apower supply switch 109 described later and an alarm lamp 119 describedlater are provided. Also, on the operations panel 1150, a sound outputsurface 1151 including plural sound output holes formed for passingthrough an output sound from a speaker 115 provided in the inside of thehousing 1100 described later is provided. Further, on the top surface ofthe housing 1100 on an area near a left side wall 1140, a holding hole1160 as a depression for receiving the arm 1200 and the camera housing1300 is provided. On the right side wall 1130 of the housing 1100,plural connection holes (1132 a through 1132 c) are provided forelectrically connecting cables to an external apparatus connecting I/F118 described later. On the other hand, on the left side wall 1140 ofthe housing 1100, a connection hole (not shown) is provided forelectrically connecting a cable 100 c for the display device 100 to theexternal apparatus connecting I/F 118.

It is noted that hereinafter, in a case where any operating button ofthe operating buttons (108 a through 108 e) is referred to, this will bereferred to as an “operating button 108”. Similarly, in a case where anyconnection hole of the connection holes (1132 a through 1132 e) isreferred to, this will be referred to as a “connection hole” 1132.

The arm 1200 is mounted on the housing 1100 via a torque hinge 1210, andis configured to be able to rotate vertically in a range of a tilt angleθ1 of 135° with respect to the housing 1100. FIG. 2 shows a state wherethe tilt angle θ1 is 90°.

The camera 112 is provided in the inside of the camera housing 1300, andthe user, a document, a room and so forth can be photographed. Further,a torque hinge 1310 is formed in the camera housing 1300. The camerahousing 1300 is mounted on the arm 1200 via the torque hinge 1310, andthus a configuration is provided such that the camera housing 1300 canbe rotated up and down (vertically), and left and right (horizontally),in a range of a pan angle θ2 of ±180° and in a range of a tilt angle θ3of ±45°, where FIG. 2 shows a state of 0°.

It is noted that each of the relay apparatuses 30, the communicationmanagement system 50, the arrangement information providing system 80and the program providing system 90 has an external appearance the sameas that of a common server computer. Therefore, description of theexternal appearances thereof will be omitted.

FIG. 3 shows a hardware configuration of the communication terminal 10according to the first embodiment. As shown in FIG. 3, the communicationterminal 10 according to the first embodiment includes a CPU (CentralProcessing Unit) 101 controlling operations of the entirety of thecommunication terminal 10; a ROM (Read Only Memory) 102 storing aprogram for the communication terminal 10; a RAM (Random Access Memory)103 used as a work area of the CPU 101; and a flash memory 104 storing aprogram for the communication terminal and various data such as imagedata and voice data. The communication terminal 10 according to thefirst embodiment further includes a SSD (Solid State Drive) 105controlling reading and writing various data from and to the flashmemory 104 according to the control of the CPU 101; a medium drive 107controlling reading and writing (recording) data from and to a recordingmedium 106 such as a flash memory; the operating buttons 108 operated bythe user in a case where the user selects a destination of thecommunication terminal 10 or so; a power supply switch 109 for switchingon and off of the power supply in the communication terminal 10; anetwork I/F (InterFace) 111 for transmitting data using thecommunication network 2; the camera 112 photographing an object andobtaining image data according to the control of the CPU 101; an imagingdevice I/F 113 controlling driving the camera 112; the microphone 114inputting a voice; the speaker 115 outputting a voice; a voiceinput/output I/F 116 processing input and output of voice signals fromthe microphone 114 and to the speaker 115 according to the control ofthe CPU 101; a display I/F 117 transmitting image data to the displaydevice 100 provided in the outside according to the control of the CPU101; the external apparatus connecting I/F 118 for connecting varioustypes of external apparatuses; the alarm lamp 119 reporting unusualconditions of various functions of the communication terminal 10; and abus line 110 such as an address bus, a data bus and so forth forelectrically connecting the above-mentioned various elements/componentsas shown in FIG. 3.

It is noted that the recording medium 106 is configured to be detachablewith respect to the communication terminal 10. Further, as long as it isa non-volatile memory for which reading and writing are carried outaccording to the control of the CPU 101, the flash memory 104 is notlimiting, and instead, an EEPROM (Electrically Erasable ProgrammableROM) or such may be used. Further, the camera 112 is a solid state imagesensing device which converts light into electric charges, and obtainsan electronic signal from an image of an object. As the camera 112, aCCD (Charge Coupled Device), a CMOS (Complementary Metal OxideSemiconductor) device or such may be used. Further, the display device100 is made of a liquid crystal or an organic electroluminescence (EL)material or such displaying an image of an object, icons used foroperations by the user or such.

Further, the program for the communication terminal 10 may be recordedin a form of a file in an installable or executable type in a computerreadable information recording medium, such as the recording medium 106,and may then be circulated.

FIG. 4 shows a hardware configuration of the communication managementsystem 50 according to the first embodiment. The communicationmanagement system 50 includes a CPU 201 controlling operations of theentirety of the communication management system 50; a ROM 202 storing aprogram for communication management (i.e., for the communicationmanagement system 50); a RAM 203 used as a work area of the CPU 201; aHD (hard disk) 204 storing various data; an HDD (Hard Disk Drive) 205controlling reading and writing of the various data from and to the HD204 according to the control of the CPU 201; a medium drive 207controlling reading and writing (recording) of data from and to arecording medium 206 such as a flash memory; a display device 200displaying various sorts of information such as a cursor, a menu, awindow, characters/letters (text) or images; a display device I/F 208; anetwork I/F 209 for carrying out data transmission using thecommunication network 2; a keyboard 211 including plural keys for theuser to input characters/letters, numerical values, various instructionsand so forth; a mouse 212 for the user to select or start execution ofvarious instructions, select a target to process, move the cursor or so;a CD-ROM drive 214 controlling reading various data from a CD-ROM(Compact Disc Read Only Memory) 213 as an example of a detachablerecording medium; an external apparatus I/F 215 transmitting andreceiving various sorts of data to and from an external apparatus; and abus line 210 such as an address bus, a data bus and so forthelectrically connecting the above-mentioned respectiveelements/components together as shown in FIG. 4.

It is noted that the program for communication management may berecorded in a computer readable recording medium such as theabove-mentioned recording medium 206, CD-ROM 213 or such in a form of afile of an installable type or an executable type, and then becirculated.

Further, the relay apparatus 30 has the same hardware configuration asthat of the communication management apparatus 50, and therefore, thedescription thereof will be omitted. However, in the ROM 202 of therelay apparatus 30, a program for the relay apparatus for controllingthe relay apparatus 30 is recorded. Also in this case, the program forthe relay apparatus may be recorded in a computer readable recordingmedium such as the recording medium 206, a CD-ROM 213 or such in a formof a file of an installable type or an executable type, and becirculated.

The external input apparatus 40 has the same hardware configuration asthat of the communication management apparatus 50, and therefore, thedescription thereof will be omitted. However, in the ROM 202 of theexternal input apparatus 40, a program for material management forcontrolling the external input apparatus 40 is recorded. Also in thiscase, the program for material management may be recorded in a computerreadable recording medium such as the recording medium 206 or the CD-ROM213 in a form of a file of an installable type or an executable type,and be circulated.

Further, each of the arrangement information providing system 80 and theprogram providing system 90 has the same hardware configuration as thatof the communication management apparatus 50, and therefore, thedescription thereof will be omitted. However, in the ROM 202 of thearrangement information providing system 80, a program for providingarrangement information is recorded. Similarly, in the ROM 202 of theprogram providing system 90, a program for providing programs isrecorded. Also in this case, each of these programs for providingarrangement information and providing programs may be recorded in acomputer readable recording medium such as the recording medium 206, theCD-ROM 213 or such in a form of a file of an installable type or anexecutable type, and be circulated.

It is noted that each of the above-mentioned programs may be recorded ina computer readable recording medium such as a CD-R (Compact DiscRecordable), a DVD (Digital Versatile Disk) or a Blu-ray Disc as anotherexample of the above-mentioned detachable recording medium, and then beprovided.

Functional Configuration of First Embodiment

Next, a functional configuration of the first embodiment will bedescribed. FIGS. 5A and 5B show a functional block diagram of each ofthe terminals, the apparatuses and the system included in thecommunication system 1 according to the first embodiment. In FIGS. 5Aand 5B, the communication terminal 10, the relay apparatus 30 and thecommunication management system 50 are connected together by thecommunication network 2 in such a manner that data communication can bemutually carried out. The external input apparatus 40 is connected withthe communication terminal 10 in such a manner that data communicationcan be mutually carried out. FIGS. 6A, 6B and 6C illustrate imagequality according to the first embodiment. FIG. 7 shows a concept of anexample of a change quality management table. FIG. 8 shows a concept ofan example of a relay apparatus management table. FIG. 9 shows a conceptof an example of a terminal authentication management table. FIG. 10shows a concept of an example of a terminal management table. FIG. 11shows a concept of an example of a destination list management table.FIG. 12 shows a concept of an example of a session management table.FIG. 13 shows a concept of an example of an address priority managementtable. FIG. 14 shows a concept of an example of a transmission ratepriority management table. FIG. 15 shows a concept of an example of aquality management table. FIG. 24 shows a concept of an arrangementinformation management table stored by a storage part. FIG. 25 shows aconcept of an area management table stored by the storage part. FIGS.26A and 26B show examples of screen pages displayed based on arrangementinformation. FIG. 27 shows a functional block diagram of the externalinput apparatus 40. FIG. 28 shows a concept of a material managementtable.

<Functional Configuration of Communication Terminal>

The communication terminal 10 includes a transmission/reception part 11,an operation input reception part 12, a login request part 13, aphotographing part 14 a, a display control part 14 b, an arrangementinformation selection part 14 c, an area determination part 14 d, avoice input part 15 a, a voice output part 15 b, a finally narrowingdown part (or selection processing part) 16, a delay detection part 17,an external information transmission/reception part 18, and astoring/reading processing part 19. The respective parts correspond tofunctions or functioning parts realized as a result of the respectiveelements/components shown in FIG. 3 operating according to instructionsgiven by the CPU 101 that operates according to the program for thecommunication terminal stored in the ROM 102. Further, the communicationterminal 10 has a storage part 1000 built by any one of the ROM 102, RAM103 and the SSD 105 shown in FIG. 3.

(Arrangement Information Management Table)

In the storage part 1000, an arrangement information management DB(DataBase) 1001 including an arrangement information management tablesuch as that shown in FIG. 24 is stored. In the arrangement informationmanagement table, in association with an arrangement information No.,arrangement information indicating data (information) to be displayed ateach area on the screen and a share flag that is reception informationindicating whether the transmission/reception part 11 has receiveddisplay data are stored. As shown in the example of FIG. 24, withrespect to the arrangement information No. “1”, the arrangementinformation “VIEW_MULTI” and the share flag “UNSHARED” are stored.

(Area Management Table)

In the storage part 1000, an area management DB 1002 including an areamanagement table such as that shown in FIG. 25 is stored. In the areamanagement table, in association with the arrangement information, datato be displayed at each area on the screen is stored. As shown in theexample of FIG. 25, the area management table shows that in a case wherethe arrangement information is “SHARED_MULTI”, received display data isdisplayed at an “area 1”; image data obtained from photographing by thephotographing part 14 a of the communication terminal 10 aa which is acommunication terminal at the other end of the communication isdisplayed at an “area 2”; and image data obtained from photographing bythe photographing part 14 a of the communication terminal 10 db which isthe own terminal is displayed at an “area 3”. Further, the areamanagement table shows that in a case where the arrangement informationis “VIEW_MULTI”, image data obtained from photographing by thephotographing part 14 a of the communication terminal 10 aa which is thecommunication terminal at the other end of the communication isdisplayed at the “area 1”; image data obtained from photographing by thephotographing part 14 a of the communication terminal 10 db which is theown terminal is displayed at the “area 2”; and nothing is displayed atthe “area 3”. The areas 1, 2 and 3 are determined by sizes and positionson the screen, respectively. Examples shown in FIGS. 26A and 26B showinformation defining areas on the screen to display respective items ofinformation. In FIG. 26A, the screen in the case where the arrangementinformation is “SHARED_MULTI” is shown, and in FIG. 26B, the screen inthe case where the arrangement information is “VIEW_MULTI” is shown. Asshown in FIGS. 26A and 26B, the area at the left side is referred to asthe area 1, the area at the top right is referred to as the area 2, andthe area at the meddle right is referred to as the area 3.

It is noted that in the case of the arrangement information“VIEW_MULTI”, at the area 1, image data to which the participant is topay attention may be displayed, and, thus, image data photographed bythe communication terminal on the speaking side may be displayed. Inorder to determine this image data, in a case where the voice input part15 a of each communication terminal 10 has detected “speaking” based onthe voice level, the display control part 14 b transfers the terminal IDof the corresponding communication terminal 10 to the relay apparatus30. The relay apparatus 30 delivers the transferred terminal ID to therespective communication terminals 10 in a broadcast manner, and thus,each of the delivered-to communication terminals can determine imagedata to be display at the area 1. It is noted that the display controlpart 14 b may transmit the terminal ID of the correspondingcommunication terminal 10 to the communication management system 50instead of the relay apparatus 30. Further, in this case, image data onthe not-speaking side is displayed at the area 2 (see FIG. 26B).

The display data means information indicating a screen page displayed onthe display device 400 of the external input apparatus 40 connected tothe communication terminal 10. It is noted that this screen page is ascreen page that displays material data created by using documentcomposition software, spreadsheet software or such and managed, andthus, is distinguished from image data obtained from photographing bythe camera 112 shown in FIG. 3.

(Detailed Functional Configuration of Communication Terminal)

Next, the respective parts of the communication terminal 10 will bedescribed in detail. The transmission/reception part 11 of thecommunication terminal 10 is realized by the network I/F 111 shown inFIG. 3, and carries out transmission and reception of various sorts ofdata (information) with another communication terminal, apparatus orsystem via the communication network 2. The operation input receptionpart 12 is realized by the operating buttons 108 and the power supplyswitch 109 shown in FIG. 3, and receives various sorts of inputoperations carried out by the user. For example, when the user turns onthe power supply switch 109, the operation input reception part 12 shownin FIG. 5B turns on the power supply in the communication terminal 10 byreceiving the user's operation of turning on the power supply switch109. The login request part 13 is realized by instructions from the CPU101 shown in FIG. 3, and automatically transmits login requestinformation indicating to request login and an IP address of thecommunication terminal 10 at the current time to the communicationmanagement system 50 via the communication network 2 from thetransmission/reception part 11 in response to receiving the user'soperation of turning on the power supply switch 109.

The photographing part 14 a is realized by the camera 112 and theimaging device I/F 113 shown in FIG. 3, photographs an object, andoutputs image data thus obtained from the photographing. The displaycontrol part 14 b is realized by the display I/F 117 shown in FIG. 3,and carries out control to display image data and display data on thedisplay device 100 provided on the outside of the communication terminal10.

The arrangement information selection part 14 c selects arrangementinformation based on the reception information indicating whether thetransmission/reception part 11 has received the display data. The areadetermination part 14 d determines areas to display each image data setand the display data based on the arrangement information selected bythe arrangement information selection part 14 c.

The voice input part 15 a is realized by the microphone 114 and thevoice input/output I/F 116 shown in FIG. 3, inputs a voice sound of theuser, and converts the input voice sound into a voice signal to obtainvoice data corresponding to the voice signal. The voice output part 15 bis realized by the speaker 115 and the voice input/output I/F 116 shownin FIG. 3, converts a voice signal corresponding to voice data into avoice, and outputs the voice.

The finally narrowing down part (or selection processing part) 16carries out a finally narrowing down process of finally narrowing downplural relay apparatuses 30 to select a single relay apparatus 30, andfor this purpose, according to instructions from the CPU 101 shown inFIG. 3, realizes a measurement part 16 a, a calculation part 16 b and afinal selection part (or selection part) 16 c. The measurement part 16 ameasures a reception date and time when advance transmission information(described later) is received by the transmission/reception part 11 foreach of advance transmission information sets received by thetransmission/reception part 11. The calculation part 16 b calculates,for each of the advance transmission information sets for which themeasurement part 16 a has measured the reception date and time, arequired time (T) from the transmission of the advance transmissioninformation to the reception thereof based on the difference between themeasured reception date and time and the transmission date and timeincluded in each of the received advance transmission information sets.The final selection part 16 c selects the relay apparatus 30 by whichthe advance transmission information has been relayed having theshortest required time (T) from among the required times (T) calculatedby the calculation part 16 b, and thus, finally selects the single relayapparatus 30.

The delay detection part 17 is realized by instructions from the CPU 101shown in FIG. 3, and detects a delay time (ms) of image data, voice dataor display data sent from another communication terminal 10 via therelay apparatus 30. Further, the external informationtransmission/reception part 18 transmits and receives data to and froman external apparatus such as the external input apparatus 40 using theexternal input apparatus I/F 215.

The storing/reading processing part 19 is realized by the SSD 105 shownin FIG. 3, stores various sorts of data in the storage part 1000, andreads various sorts of data stored in the storage part 1000. In thestorage part 1000, terminal IDs (Identifications) for identifying thecommunication terminals 10, passwords, image data, voice data and soforth are stored.

It is noted that each of the terminal ID and a relay apparatus IDdescribed later according to the first embodiment indicatesidentification information such as a language, a character/letter, asign or various sorts of marks used for uniquely identifying arespective one of the communication terminal 10 and the relay apparatus30. Further, each of the terminal ID and the relay apparatus ID may beidentification information that is a combination of at least two of theabove-mentioned language, character/letter, sign and various sorts ofmarks. Further, hereinafter, the communication terminal 10 as a requestsource which request a start of a TV conference will be referred to as a“request source terminal 10A” and the communication terminal 10 as adestination as a request destination will be referred to as a“destination terminal 10B”.

<Functional Configuration of Relay Apparatus>

Functions and respective parts of the relay apparatus 30 will now bedescribed. The relay apparatus 30 includes a transmission/reception part31, a state detection part 32, a data quality determination part 33, achange quality management part 34, a data quality changing part 35 and astoring/reading processing part 39 (see FIG. 5B). These respective partscorrespond to functions or functioning parts realized as a result of therespective elements/components shown in FIG. 4 operating according toinstructions given by the CPU 201 that operates according to the programstored in the ROM 202. Further, the relay apparatus 30 has a storagepart 3000 comprising any one of the ROM 202, the RAM 203 and the HDD 205shown in FIG. 4.

(Change Quality Management Table)

In the storage part 3000, a change quality management DB (DataBase) 3001including the change quality management table such as that shown in FIG.7 is stored. In the change quality management table, the IP address ofthe communication terminal 10 as a relay destination (or simply adestination) of image data and the image quality of the image relayed bythe relay apparatus 30 to the relay destination are associated with oneanother and are managed.

Resolution of image data used in the first embodiment will now bedescribed. As shown in FIG. 6A, an image of low resolution as a baseimage has a configuration of 160 pixels (in a horizontal direction) by120 pixels (in a vertical direction). As shown in FIG. 6B, an image ofmedium resolution has a configuration of 320 pixels (in the horizontaldirection) by 240 pixels (in the vertical direction). As shown in FIG.6C, an image of high resolution has a configuration of 640 pixels (inthe horizontal direction) by 480 pixels (in the vertical direction). Ina case where a narrow band path is used, image data of low image qualityincluding only image data of low resolution as a base image is relayed.In a case where a band is relatively wide, image data of medium imagequality including image data of low resolution as a base image and imagedata of medium resolution is relayed. In a case where a band is verywide, image data of high image quality including image data of lowresolution as a base image, image data of medium resolution and imagedata of high resolution is relayed. For example, in the change qualitymanagement table shown in FIG. 7, in a case where the relay apparatus 30relays image data to the destination terminal 10 db of the IP address“1.3.2.4”, the image quality (quality of an image) of the image data is“high quality” (high image quality).

(Detailed Functional Configuration of Relay Apparatus)

Next, the functional configuration of the relay apparatus 30 will bedescribed in detail. It is noted that below, along with describing thefunctional configuration of the relay apparatus 30, relationships withmain elements/components, from among those shown in FIG. 4, used forrealizing the functional configuration of the relay apparatus 30 will bealso described.

The transmission/reception part 31 of the relay apparatus 30 shown inFIG. 5B is realized by the network I/F 209 shown in FIG. 4, and carriesout transmission and reception of various sorts of data (information)with another communication terminal, apparatus or system via thecommunication network 2. The state detection part 32 is realized byinstructions from the CPU 201, and detects an operating state of therelay apparatus 30 having this state detection part 32. The operatingstate may be a state of “on-line”, “off-line”, or “in failure”.

The data quality determination part 33 is realized by instructions fromthe CPU 201, searches the change quality management DB 3001 (see FIG. 7)using the IP address of the destination terminal 10B as a search key,extracts the image quality of the corresponding image data to berelayed, and thus recognizes the image quality of the image data to berelayed. The change quality management part 34 is realized byinstructions from the CPU 201, and changes the contents of the changequality management DB 3001, if necessary, based on quality information(described later) which is sent from the communication management system50. For example, a case is supposed in which a TV conference is beingconducted between a request source terminal (10 aa) having the terminalID “01aa” and a destination terminal (10 db) having the terminal ID“01db” where image data of high image quality is mutually transmitted,and a delay in receiving the image data occurs in the destinationterminal (10 db) because another request source terminal (10 bb) andanother destination terminal (10 ca) have started another TV conferenceusing the communication network 2, for example. In such a case, therelay apparatus 30 needs to reduce the image quality of the image data,being relayed by the relay apparatus 30 itself, from the high imagequality to the medium image quality. In such a case, based on thequality information indicating this medium image quality, the contentsof the change quality management DB 3001 are changed so that the imagequality of the image data which the relay apparatus 30 is relaying isreduced from the high image quality to the medium image quality.

The data quality changing part 35 is realized by instructions from theCPU 201, and changes the image quality of the image data sent from thetransmission source terminal 10 based on the contents of the changequality management DB 3001 changed as mentioned above. Thestoring/reading processing part 39 is realized by the HDD 205 shown inFIG. 4, and carries out processing of recording various sorts of data inthe storage part 3000 and reading various sorts of data stored in thestorage part 3000.

<Functional Configuration of External Input Apparatus>

The external input apparatus 40 includes, as shown in FIG. 27, atransmission/reception part 41, an operation input reception part 42, adisplay control part 43, a display data obtaining part 44, a displaydata transmission part 44A and a storing/reading processing part 49.These respective parts are functions or functional parts shown in FIG. 4realized as a result of the CPU 201 operating according to the programstored in the ROM 202. Further, the external input apparatus 40 has astorage part 4000 comprising any one of the ROM 202, the RAM 203 and theHDD 205.

(Material Management Table)

In the storage part 4000, a material management DB (DataBase) 4001including a material management table such as that shown in FIG. 28 isstored. In the material management table, a material name that is a nameof material; a size indicating a size of the material; an applicationname indicating a name of application software used for creating thematerial; an updated date and time at which the material has beenupdated most recently; and material data are associated with each otherand are managed. For example, in the material management table shown inFIG. 28, it is shown that material name “x x project book”, size “182kB”, application name “document composition software”, updated date andtime “2010/03/19” and material data “x x project book.xls” areassociated with each other.

<Detailed Functional Configuration of External Input Apparatus>

Next, each part of the external input apparatus 40 will be described indetail. The transmission/reception part 41 of the external inputapparatus 40 is realized by the external input apparatus I/F 215 shownin FIG. 4, and transmits and receives various sorts of data(information) to and from the communication terminal 10. The operationinput reception part 42 receives an operation of the user for inputtingan instruction or such. The display control part 43 displays on thedisplay device 400 an image read out by the storing/reading processingpart 49 described later. The display data obtaining part 44 obtainsdisplay data of a screen page which is displayed by the external inputapparatus 40. The display data transmission part 44A transmits thedisplay data obtained by the display data obtaining part 44 to thecommunication terminal 10.

The storing/reading processing part 49 is realized by the HDD 205 shownin FIG. 4, stores various sorts of data in the storage part 4000 andreads the various sorts of data stored in the storage part 4000. In thestorage part 4000, the material data and so forth to be shared byparticipants of a TV conference is stored.

<Functional Configuration of Communication Management System>

Next, functions or functional parts of the communication managementsystem 50 will be described. As shown in FIG. 5A, the communicationmanagement system 50 includes a transmission/reception part 51, aterminal authentication part 52, a state management part 53, a terminalextraction part 54, a terminal state obtaining part 55, a narrowing downpart 56, a session management part 57, a quality determination part 58,a storing/reading processing part 59 and a delay time management part60. These respective parts correspond to functions or functioning partsrealized as a result of the respective elements/components shown in FIG.4 operating according to instructions given by the CPU 201 that operatesaccording to the program for communication management stored in the ROM202. Further, the communication management system 50 has a storage part5000 comprising any one of the ROM 202, the RAM 203 and the HD 204 shownin FIG. 4.

(Relay Apparatus Management Table)

In the storage part 5000, a relay apparatus management DB 5001 includingthe relay apparatus management table such as that shown in FIG. 8 isstored. In the relay apparatus management table, for each of the relayapparatuses 30, the relay apparatus ID of the relay apparatus 30, theoperating state of the relay apparatus 30, the reception date and timewhen the state information indicating the operating state has beenreceived by the communication management system 50, the IP address ofthe relay apparatus 30 and the maximum data transmission rate (Mbps) inthe relay apparatus 30 are associated with each other and are managed.For example, in the relay apparatus management table shown in FIG. 8, itis indicated that the relay apparatus 30 a having the relay apparatus ID“111a” has the operating state of “on-line”, the date and time when thestate information has been received in the communication managementsystem 50 are “November 10, 2009, 13:00”, the IP address of the relayapparatus 30 a is “1.2.1.2”, and the maximum data transmission rate inthe relay apparatus 30 a is 100 Mbps.

(Terminal Authentication Management Table)

Further, in the storage part 5000, a terminal authentication managementDB 5002 including the terminal authentication management table such asthat shown in FIG. 9 is stored. In the terminal authenticationmanagement table, respective passwords are associated with the terminalIDs of all the terminals 10 managed by the communication managementsystem 50 and are managed. For example, in the terminal authenticationmanagement table shown in FIG. 9, it is indicated that the terminal IDof the terminal 10 aa is “01aa”, and the password is “aaaa”.

(Terminal Management Table)

Further, in the storage part 5000, a terminal management DB 5003including the terminal management table such as that shown in FIG. 10 isstored. In the terminal management table, for the terminal ID of each ofthe communication terminals 10, the operating state of the communicationterminal 10, the reception date and time when login request information(described later) has been received by the communication managementsystem 50, and the IP address of the communication terminal 10 areassociated with each other and are managed. For example, in the terminalmanagement table shown in FIG. 10, it is indicated that thecommunication terminal 10 aa having the terminal ID “01aa” has theoperating state “on-line (telephone call possible)”, the reception dateand time when the login request information has been received in thecommunication management system 50 is “November 10, 2009, 13:40”, andthe IP address of the communication terminal 10 aa is “1.2.1.3”.

(Destination List Management Table)

Further, in the storage part 5000, a destination list management DB 5004including the destination list management table such as that shown inFIG. 11 is stored. In the destination list management table, for theterminal ID of a request source terminal 10A which requests to start aTV conference, all of the terminal IDs of the destination terminals 10Bregistered as candidates for a destination terminal 10B are associatedand are managed. For example, in the destination list management tableshown in FIG. 11, it is indicated that the candidates for a destinationterminal 10B for which the request source terminal (10 aa) can requestto start of a TV conference are three, i.e., the terminal 10 ab havingthe terminal ID “01ab”, the terminal 10 ba having the terminal ID “01ba”and the terminal 10 db having the terminal ID “01db”. The candidates fora destination terminal 10B are updated as a result of addition ordeletion according to a request of addition or deletion from any requestsource terminal to the communication management system 50.

(Session Management Table)

Further, in the storage part 5000, a session management DB 5005including the session management table such as that shown in FIG. 12 isstored. In the session management table, for each of selection sessionIDs used for carrying out a session (selection session) for selectingthe relay apparatus 30, the relay apparatus ID of the relay apparatus 30(used for relaying image data and voice data), the terminal ID of therequest source terminal 10A, the terminal ID of the destination terminal10B, a delay time (ms) (of reception when the image data is received atthe destination terminal 10B), and the reception date and time (at whichdelay information indicating the delay time has been transmitted fromthe destination terminal 10B and received by the communicationmanagement system 50) are associated with each other and managed. It isnoted that the delay time (ms) means a delay time of reception whenimage data is received at the destination terminal 10B. The receptiondate and time means reception date and time when delay informationindicating the delay time has been sent from the destination terminal10B and received in the communication management system 50.

For example, in the session management table shown in FIG. 12, it isindicated that the relay apparatus 30 a (the relay apparatus ID “111a”)selected by the session executed using the selection session ID “se1” isrelaying image data and voice data between the request source terminal(10 aa) of the terminal ID “01aa” and the destination terminal (10 db)of the terminal ID “01db”, and the delay time of the image data is 200ms at the time of “November 10, 2009, 14:00” at the destination terminal(the terminal 10 db).

It is noted that in a case where a TV conference is carried out betweentwo communication terminals 10, the reception date and time of the delayinformation may be managed based on the delay information sent from therequest source terminal 10A instead of the destination terminal 10B.However, in a case where a TV conference is conducted between three ormore communication terminals 10, the reception date and time of thedelay information is managed based on the delay information sent fromthe communication terminal which is receiving the image data and thevoice data.

(Address Priority Management Table)

Further, in the storage part 5000, a priority management DB 5006including an address priority management table such as that shown inFIG. 13 is stored. In the address priority management table, differencesin four values included in dot addresses of an IP address according tothe common IPv4 are associated with a point count of address priority insuch a manner that the point count of address priority becomes higher asthe number of same dot addresses is larger. For example, in the addresspriority management table shown in FIG. 13, in a case (on first line)where the IP address has three values of the dot addresses from thehighest order toward the lowest order which are the same as each other(i.e., “same.same.same.diff” in FIG. 13), the point count of the addresspriority is “5”. In a case (on second line) where the IP address has twovalues of the dot addresses from the highest order toward the lowestorder which are the same as each other (i.e., “same.same.diff.-”), thepoint count of the address priority is “3”. In this case, whether thevalue at the lowest order is the same or different is not relevant tothe priority. In a case (on third line) where the IP address has thevalue of the dot address at the highest order is the same and the secondorder value is different (i.e., “same.diff.-.-”), the point count of theaddress priority is “1”. In this case, whether the values at the thirdand last orders from the highest order are the same or different is notrelevant to the priority. In a case (on last line) where the IP addresshas the value of the dot address at the highest order is different(i.e., “diff.-.-.-”), the point count of the address priority is “0”. Inthis case, whether the values at the second, third and last orders fromthe highest order are the same or different is not relevant to thepriority.

(Transmission Rate Priority Management Table)

Further, in the priority management DB 5006 stored in the storage part5000, a transmission rate management table such as that shown in FIG. 14is also included. In the transmission rate priority management table,the maximum data transmission rate (Mbps) in the relay apparatus 30 isassociated with the point count of priority in such a manner that as themaximum transmission rate is larger, the point count of transmissionrate priority becomes higher. For example, in the transmission ratepriority management table shown in FIG. 14, in a case (on first line)where the maximum transmission rate is equal to 1000 Mbps or more, thepoint count of the transmission rate priority is “5”. In a case (onsecond line) where the maximum transmission rate is equal to more than100 Mbps and less than 1000 Mbps, the point count of the transmissionrate priority is “3”. In a case (on third line) where the maximumtransmission rate is equal to more than 10 Mbps and less than 100 Mbps,the point count of the transmission rate priority is “1”. In a case (onlast line) where the maximum transmission rate is less than 10 Mbps, thepoint count of the transmission rate priority is “0”.

(Quality Management Table)

Further, in the storage part 5000, a quality management DB 5007including a quality management table such as that shown in FIG. 15 isstored. In the quality management table, image quality (quality of animage) of the image data to be relayed by the relay apparatus 30 isassociated with the delay time (ms) of the image data in the requestsource terminal 10A or the destination terminal 10B and is managed. Itis noted that, as shown in FIG. 15, in a case where the delay time ismore than “500 ms”, the communication is considered to be interrupted(“INTERRUPTION”) since the delay is too much.

(Detailed Functional Configuration of Communication Management System)

Next, the functional configuration of the communication managementsystem 50 will be described in detail. It is noted that below along withdescribing the functional configuration of the communication managementsystem 50, relationships with main elements/components, from among thoseshown in FIG. 5A, used for realizing the functional configuration of thecommunication management system 50 will be also described.

The transmission/reception part 51 is realized by the network I/F 209shown in FIG. 4, and carries out transmission and reception of varioussorts of data (information) with another terminal, apparatus or systemvia the communication network 2. The terminal authentication part 52searches the terminal authentication management DB 5002 of the storagepart 5000 using the terminal ID and the password included in the loginrequest information received via the transmission/reception part 51 assearch keys, and carries out authentication of the communicationterminal 10 by determining whether the same terminal ID and password aremanaged in the terminal authentication management DB 5002. The statemanagement part 53, for the purpose of managing the operating state of arequest source terminal 10A which has requested to login, stores theterminal ID of the request source terminal 10A, the operating state ofthe request source terminal 10A, the reception date and time when thelogin request information has been received in the communicationmanagement system 50 and the IP address of the request source terminalin the terminal management DB 5003 (i.e., the terminal management table)(see FIG. 11) in a manner of associating them with each other andmanages them.

The terminal extraction part 54 searches the destination list managementDB 5004 (i.e., the destination list management table) (see FIG. 11)using the terminal ID of a request source terminal 10A which hasrequested to login as a search key, reads the terminal IDs as candidatesfor a destination terminal 10B which can carry out communication (orremote communication) with the request source terminal 10A, and extractsthe terminal IDs. Further, the terminal extraction part 54 searches thedestination list management DB 5004 (i.e., the destination listmanagement table) using the terminal ID of a request source terminal 10Awhich has requested to login as a search key, and extracts also theterminal IDs of the other request source terminals 10A each of which hasregistered the terminal ID of the request source terminal 10A as acandidate for a destination terminal 10B.

The terminal state obtaining part 55 searches the terminal management DB5003 (i.e., the terminal management table) (see FIG. 10) using theterminal IDs of candidates for a destination terminal 10B extracted bythe terminal extraction part 54 as search keys, and reads the operatingstate for each of the terminal IDs extracted by the terminal extractionpart 54. Thereby, the terminal state obtaining part 55 can obtain theoperating states of the candidates for a destination terminal 10B whichcan carry out communication (or remote communication) with the requestsource terminal 10A having requested to login. Further, the terminalstate obtaining part 55 searches the terminal management DB 5003 (i.e.,the terminal management table) (see FIG. 10) using the terminal ID ofthe request source terminal having requested to login, and obtains theoperating state of the request source terminal 10A having requested tologin.

The narrowing down part (or preliminary narrowing down part) 56, for thepurpose of assisting in a finally narrowing down process of finallynarrowing down plural relay apparatuses 30 to select a single relayapparatus 30, and for carrying out a preliminary narrowing down processbefore the finally narrowing down process, has a selection session IDgeneration part 56 a, a terminal ID obtaining part 56 b, a preliminaryselection part 56 c, and a priority determination part 56 d. Thereamong,the selection session ID generation part 56 a generates a selectionsession ID to be used for carrying out a session (selection session) forselecting relay apparatuses 30. The terminal ID obtaining part 56 bsearches the terminal management table (of the terminal management DB5003) (see FIG. 10) based on the terminal ID of the request sourceterminal 10A and the terminal ID of the destination terminal 10B fromstart request information sent from the request source terminal 10A, andextracts the corresponding IP addresses of the respective communicationterminals 10. The preliminary selection part 56 c selects the relayapparatus IDs of the relay apparatuses having the operating states“on-line” from among the relay apparatuses 30 managed in the relayapparatus management DB 5001 (i.e., the relay apparatus managementtable) (see FIG. 8), and thus, selects the relay apparatuses 30.

Further, based on the terminal ID of the request source terminal 10A andthe terminal ID of the destination terminal 10B obtained by the terminalID obtaining part 56 b, the preliminary selection part 56 c searches therelay apparatus management table (of the relay apparatus management DB5001) (see FIG. 8), and thus, determines whether for each of the dotaddresses of the IP addresses of the thus-selected relay apparatuses 30,the respective values of each of the dot addresses of the IP addressesof the above-mentioned request source terminal 10A and the destinationterminal 10B are the same. Further, the preliminary selection part 56 cfurther selects the relay apparatuses 30 by selecting the top two relayapparatuses 30 having the highest and second highest point counts ofpriority, respectively, among total point counts obtained from totalingthe points of the address priority and the point counts of thetransmission rate priority. It is noted that as described later withreference to FIG. 19, each of the point counts of the address priorityused for the totaling is one higher with respect to the correspondingcommunication terminals 10.

It is noted that according to the first embodiment, the top two relayapparatuses 30 having the highest and second highest point counts areselected as mentioned above. However, this way is not limiting, and itis also possible to select the top three or more relay apparatuses 30having the highest, second highest and third highest point counts ormore when the more relay apparatuses 30 are to be selected fromnarrowing down.

The priority determination part 56 d reads the address priory managementtable (of the priority management DB 5006) (see FIG. 13) and determinesthe point counts of the address priority for each of the relayapparatuses 30 for which the preliminary selection part 56 c hasdetermined whether for the dot addresses of the IP addresses of thethus-selected relay apparatuses 30, the respective values of the dotaddresses of the IP addresses of the above-mentioned request sourceterminal 10A and the destination terminal 10B are the same. Further, thepriority determination part 56 d searches the transmission rate priorymanagement table (of the priority management table DB 5006) (see FIG.14) based on the maximum data transmission rate of each of the relayapparatuses 30 managed in the relay apparatus management table (of therelay apparatus management DB 5001) (see FIG. 8), and determines thepoint count of the transmission rate priority for each of the relayapparatuses 30 having been selected as having the operating states of“on-line” as mentioned above.

The session management part 57 stores the selection session ID generatedby the selection session ID generation part 56 a, the terminal ID of therequest source terminal 10A and the terminal ID of the destinationterminal 10B in a manner of associating them with each other in thesession management table (of the session management DB 5005) (see FIG.12) of the storage part 5000, and manages them. Further, the sessionmanagement part 57 stores for each of the selection session IDs, therelay apparatus ID of the relay apparatus 30 which has been finallynarrowed down to one by the selection part 16 c of the communicationterminal 10 in the session management table (of the session managementDB 5005) (see FIG. 12) and manages them.

The quality determination part 58 searches the quality management table(of the quality management DB 5007) (see FIG. 15) using theabove-mentioned delay time as a search key, extracts the image qualityof the corresponding image data, and determines the image quality of theimage data to be relayed by the relay apparatus 30. The storing/readingprocessing part 59 is realized by the HDD 205 shown in FIG. 4, storesvarious sorts of data in the storage part 5000 and reads various sortsof data stored in the storage part 5000. The delay time management part60 searches the terminal management table (of the terminal management DB5003) (see FIG. 10) using the IP address of the above-mentioneddestination terminal 10B, extracts the corresponding terminal ID, andfurther stores the delay time indicated by the above-mentioned delayinformation at the field of delay time in the record including thethus-extracted terminal ID in the session management table (of thesession management DB 5005) (see FIG. 12), and manages it.

Processing/Operations of First Embodiment

Thus, the configuration and functions or functional parts of thecommunication system according to the first embodiment has beendescribed. Next, with reference to FIGS. 16 through 23, and FIGS. 29through 33, a processing method in the communication system 1 accordingto the first embodiment will be described. It is noted that FIG. 16 is asequence diagram showing a process of managing the informationindicating the states of the relay apparatuses 30 transmitted to thecommunication management system 50 from the corresponding relayapparatuses 30, respectively. FIG. 17 is a sequence diagram showing aprocess of a preparation stage for starting communication between pluralcommunication terminals 10. FIG. 18 is a sequence diagram showing aprocess of narrowing down the relay apparatuses 30. FIG. 19 is aflowchart showing the process of narrowing down the relay apparatuses30. FIG. 20 shows a state of calculation of point counts used whennarrowing down the relay apparatuses 30. FIG. 21 is a sequence diagramshowing the process of selecting a relay apparatus 30 by thecommunication terminal 10. FIG. 22 is a flowchart showing the process ofselecting a relay apparatus 30 by the communication terminal 10. FIG. 23is a sequence diagram showing a process of transmitting image data andvoice data between the communication terminals 10. FIG. 29 is a sequencediagram showing a process of displaying the image data and the displaydata (material data) by the communication terminal. FIG. 30 is aflowchart showing a process of determining, by the communicationterminal, arrangement information used for displaying the image data andthe display data. FIG. 31 is a sequence diagram showing a process ofstopping transmission of the display data. FIG. 32 is a flowchartshowing a process of determining the arrangement information based onoperations carried out by a user. FIG. 33 is a state transition diagramshowing a state transition of the arrangement information based onoperations carried out by the user.

First, with reference to FIG. 16, a process of managing the stateinformation indicating the states of the respective relay apparatuses 30transmitted to the communication management system 50 from therespective relay apparatuses 30 will be described. First, in each of therelay apparatuses 30, the state detection part 32 shown in FIG. 5Bperiodically detects the operating state of the relay apparatus 30 (ownapparatus) (steps S1-1 through S1-4). Then, in order to cause thecommunication management system 50 to manage the operating states of therespective relay apparatuses 30 in a real-time manner, thetransmission/reception part 31 of each of the relay apparatuses 30periodically transmits the operating state of the own apparatus to thecommunication management system 50 via the communication network 2(steps S2-1 through S2-4). The state information includes the relayapparatus ID of the corresponding relay apparatus 30, and the operatingstate detected by the state detection part 32 of the relay apparatus 30corresponding to the relay apparatus ID. It is noted that a case will besupposed where the relay apparatuses 30 a, 30 b and 30 d are operatingnormally and are in “on-line” states, respectively, while the relayapparatus 30 c is operating but some failure has occurred in a programfor carrying out a relay operation and is in a state of “off-line”.

Next, in the communication management system 50, thetransmission/reception part 51 receives the state information sent fromeach of the relay apparatuses 30, and the state information is storedand managed for each of the relay apparatus IDs in the relay apparatusmanagement table (see FIG. 8) in the storage part 5000 via thestoring/reading processing part 59 (steps S3-1 through S3-4). Thereby,as shown in FIG. 8, in the relay apparatus management table, for each ofthe relay apparatus IDs, one of operating states of “on-line”,“off-line” and “in failure” is stored and managed. At this time, foreach of the relay apparatus IDs, the reception date and time when thestate information has been received in the communication managementsystem 50 is stored and managed. It is noted that in a case where thestate information has not been sent from the relay apparatus 30, thefield area of operating state and the field area of reception date andtime on the corresponding record in the relay apparatus management tableshown in FIG. 8 become blank, or show the operating state and thereception date and time at the time of the preceding reception.

Next, with reference to FIG. 17, a process of transmission/reception ofvarious sorts of management information at a preparation stage beforestarting communication (or remote communication) between thecommunication terminal 10 aa and the terminal 10 db will be described.First, when the user turns on the power supply switch 109 shown in FIG.2, the operation input reception part 12 shown in FIG. 5B receives thepower supply turning on operation, and turns on the power supply in thecommunication terminal 10 (step S21). Then, in response to reception ofthe above-mentioned power supply turning on operation, the login requestpart 13 automatically transmits login request information indicating alogin request to the communication management system 50 via thecommunication network 2 from the transmission/reception part 11 (stepS22). The login request information includes the terminal ID foridentifying the own terminal (communication terminal 10 aa) as therequest source and the password. The terminal ID and password are datahaving been read out from the storage part 1000 via the storing/readingprocessing part 19 and having been sent to the transmission/receptionprocessing part 11. It is noted that when the login request informationis transmitted to the communication management system 50 from thecommunication terminal 10 aa, the communication management system 50that has received the login request information can obtain the IPaddress of the communication terminal 10 aa that has sent the loginrequest information.

Next, the terminal authentication part 52 in the communicationmanagement system 50 searches the terminal authentication managementtable stored in the terminal authentication management DB 5002 (see FIG.9) of the storage part 5000 using the terminal ID and the passwordincluded in the login request information having been received from thetransmission/reception part 51 as search keys, determines whether thesame terminal ID and password are managed in the terminal authenticationmanagement table, and thus, carries out authentication of thecommunication terminal by determining whether the same terminal ID andpassword are managed (step S23). In a case where it has been determinedby the terminal authentication part 52 that the login request is onesent from the communication terminal 10 having proper use authoritysince the same terminal ID and password are managed, the statemanagement part 53 stores the terminal ID of the terminal 10 aa, theoperating state thereof, the reception date and time when theabove-mentioned login request information has been received and the IPaddress of the terminal 10 aa in the terminal management table stored inthe terminal management DB 5003 (see FIG. 10) in a manner of associatingthem with each other (step S24). Thereby, in the terminal managementtable shown in FIG. 10, the operating state “on-line”, the receptiondate and time “2009.11.10.13:40” and the IP address “1.2.1.3” areassociated with the terminal ID “01aa” and are managed.

Then, the transmission/reception part 51 of the communication managementsystem 50 transmits authentication result information indicating theresult of the authentication obtained by the terminal authenticationpart 52 to the request source terminal (10 aa), having carried out theabove-mentioned login request, via the communication network 2 (stepS25). Below, description will be carried out continuously for a casewhere the terminal authentication part 52 has determined that theterminal having carried out the above-mentioned login request has theproper use authority.

The terminal extraction part 54 of the communication management system50 searches the destination list table stored in the destination listmanagement DB 5004 (see FIG. 11) using the terminal ID “01aa” of therequest source terminal (10 aa) having carried out the login request asa search key, and reads out and extracts the terminal IDs as candidatesfor a destination terminal which can carry out communication with therequest source terminal (10 aa) (step S26). Here, the terminal IDs“01ab”, “01ba” and “01db” of destination terminals (10 ab, 10 ba and 10db) corresponding to the terminal ID “01aa” of the request sourceterminal (10 aa) are extracted.

Next, the terminal state obtaining part 55 searches the terminalmanagement table stored in the terminal management DB 5003 (see FIG. 10)using these terminal IDs “01ab”, “01ba” and “01db” of the candidates fora destination terminal extracted by the terminal extraction part 54 assearch keys, reads the respective operating states “off-line”, “on-line”and “on-line” for the above-mentioned respective terminal IDs extractedby the terminal extraction part 54, and obtains the respective operatingstates of the terminals 10 ab, 10 ba and 10 db (step S27).

Next, the transmission/reception part 51 transmits destination stateinformation including the terminal IDs “01ab”, “01ba” and “01db” as thesearch keys used in step S27 and the operating states “off-line”,“on-line” and “on-line” of the respective destination terminals (10 ab,10 ba and 10 db) to the request source terminal (10 aa) via thecommunication network 2 (step S28). Thereby, the request source terminal(10 aa) can obtain the respective operating states “off-line”, “on-line”and “on-line” of the terminals 10 ab, 10 ba and 10 db at the currenttime which are the candidates for a destination terminal 10B which cancarry out communication (or remote communication) with the requestsource terminal (10 aa).

Further, the terminal extraction part 54 of the communication managementsystem 50 searches the destination list management table stored in thedestination list management DB 5004 (see FIG. 11) using the terminal ID“01aa” of the request source terminal (10 aa) having carried out thelogin request as a search key, and extracts the terminal IDs of theother request source terminals which have registered the above-mentionedrequest source terminal (10 aa) as a candidate for a destinationterminal 10B (step S29). In the destination list table shown in FIG. 11,the terminal IDs of the other request source terminals to be extractedare “01ab”, “01ba” and “01db”.

Next, the terminal state obtaining part 55 of the communicationmanagement system 50 searches the terminal state management table storedin the terminal state management DB 5003 (see FIG. 10) using theterminal ID “01aa” of the request source terminal (10 aa) having carriedout the above-mentioned login request, and obtains the operating stateof the request source terminal (10 aa) having carried out the loginrequest (step S30).

Then, the transmission/reception part 51 transmits destination stateinformation including the terminal ID “01aa” of the request sourceterminal (10 aa) and the operating state “on-line” thereof obtained instep S30 to the terminals 10 ba and 10 db which have the operating stateof “on-line” in the terminal management table stored in the terminalmanagement DB 5003 (see FIG. 10) among the terminals 10 ab, 10 ba and 10db having the terminal IDs 01ab, 01ba and 01db extracted in step S29(step S31-1, S31-2). It is noted that when the transmission/receptionpart 51 is to transmit the destination state information to theterminals 10 ba and 10 db, the transmission/reception part 51 reads theIP addresses of these terminals managed in the terminal management tableshown in FIG. 10 based on the respective terminal IDs “01ba” and “01db”.Thereby, it is possible to inform the respective destination terminals(10 ba and 10 db) which can carry out communication (or remotecommunication) with the request source terminal (10 aa) having carriedout the login request regarded as the destination, of the terminal ID“01aa” and the operating state “on-line” of the request source terminal(10 aa) having carried out the above-mentioned login request.

On the other hand, also in each of the other communication terminals 10,the same as the above-mentioned step S21, when the user turns on thepower supply switch 109 shown in FIG. 4, the operation input receptionpart 12 shown in FIG. 5B receives the power supply turning on operation,and the processes the same as those of steps S22 through S31-1 and S31-2mentioned above are carried out. Therefore, the description thereforwill be omitted.

Next, with reference to FIG. 18, the process of narrowing down the relayapparatuses 30 will be described. It is noted that according to thefirst embodiment, the request source terminal (10 aa) can carry outcommunication (or remote communication) with at least one of thecommunication terminals (10 ba and 10 db) having the operating state of“on-line” among the communication terminals 10 of the candidates for adestination. Therefore, below, description will be carried out for acase where the user of the request source terminal (10 aa) has selectedto carry out communication (or remote communication) with thedestination terminal (10 db).

First, when the user presses the operating button 108 to select thecommunication terminal 10 db, the operation input reception part 12shown in FIG. 5B receives the selection of the destination terminal asthe communication terminal 10 db or receives a request to startcommunication (or remote communication) with the communication terminal10 db (step S41). Then, the transmission/reception part 11 of therequest source terminal (10 aa) transmits start request informationincluding the terminal ID “01aa” of the terminal 10 aa and the terminalID “01db” of the destination terminal (10 db) and indicating anintention to start communication (or remote communication) to thecommunication management system 50 (step S42). Thereby, thetransmission/reception part 51 of the communication management system 50receives the start request information, and can obtain the IP address“1.2.1.3” of the request source terminal (10 aa) (transmission source).Then, the state management part 53 changes the field area of operatingstate in each of the records of the above-mentioned terminal ID “01aa”and terminal ID “01db”, respectively, in the terminal management tablestored in the terminal management table 5003 (see FIG. 10), into “on atelephone call”, based on the terminal ID “01aa” of the request sourceterminal (10 aa) and the terminal ID “01db” of the destination terminal(10 db) included in the start request information (step S43). It isnoted that in this state, although communication (or remotecommunication) (a telephone call) has not yet been started between therequest source terminal (10 aa) and the destination terminal (10 db),the request source terminal (10 aa) and the destination terminal (10 db)thus enter the state of “on a telephone call”. Therefore, when anothercommunication terminal 10 intends to carry out a telephone call with therequest source terminal (10 aa) or the destination terminal (10 db), anotification sound or display indicating the “on a telephone call” stateis output.

Next, a process of carrying out a session for selecting a relayapparatus 30 will be described with reference to steps S44 through S48(FIG. 18) and steps S61-1 through S66 (FIG. 21). First, the selectionsession ID generating part 56 a generates a selection session ID used tocarry out a session for selecting a relay apparatus 30 (step S44). Then,the session management part 57 stores the selection session ID “se1”generated in step S44, the terminal ID “01aa” of the request sourceterminal (10 aa) and the terminal ID “01db” of the destination terminal(10 db) in a manner of associating them with each other in the sessionmanagement table stored in the session management DB 5005 (see FIG. 12)of the storage part 5000 and manages them (step S45).

Next, the narrowing down part (or preliminary narrowing down part) 56 ofthe communication management system 50 carries out preliminary narrowingdown the relay apparatuses 30 for relaying communication between therequest source terminal (10 aa) and the destination terminal (10 db)based on the relay apparatus management DB 5001, the terminal managementDB 5003 and the priority management DB 5006 (step S46).

With reference to FIG. 19, the process in step S46 will now be describedin detail. First, the terminal IP address extraction part 56 b searchesthe terminal management table stored in the terminal management DB 5003(see FIG. 10) based on the terminal ID “01aa” of the request sourceterminal 10 aa and the terminal ID “01db” of the destination terminal 10db included in the start request information sent from the requestsource terminal 10 aa, and extracts the IP addresses (“1.2.1.3” and“1.3.2.4”) of the corresponding communication terminals (10 aa, 10 db)(step S46-1). Next, the preliminary selection part 56 c selects therespective relay apparatus IDs (111a, 111b, 111d) of the relayapparatuses (30 a, 30 b, 30 d) which are “on-line” from among theoperating states of the relay apparatuses 30 managed in the relayapparatus management table stored in the relay apparatus management DB5001 (see FIG. 8). Further, the preliminary selection part 56 c searchesthe relay apparatus management table stored in the relay apparatusmanagement DB 5001 (see FIG. 8) based on the IP address “1.2.1.3” of therequest source terminal 10 aa and the IP address “1.3.2.4” of thedestination terminal 10 db extracted in step S46-1, and determineswhether for the dot addresses of the respective IP addresses (“1.2.1.2”,“1.2.2.2”, “1.3.2.2”) of the relay apparatuses (30 a, 30 b, 30 d)selected in step S46-2, the dot address is the same as the dot addressesof the IP addresses (“1.2.1.3” and “1.3.2.4”) of the request sourceterminal 10 aa and the destination terminal 10 db (step S46-3).

Next, the priority determination part 56 d reads the priority managementtable stored in the priority management DB 5006 (see FIG. 13) anddetermines the point count of the address priority for each of the relayapparatuses (30 a, 30 b, 30 d) determined in step S46-3 (step S46-4).FIG. 20 shows a result of the determination. It is noted that FIG. 20shows a calculation state for the point counts of the priority used whenthe process of narrowing down the relay apparatuses 30 is carried out.In FIG. 20, for each of the relay apparatus IDs, the point counts of theaddress priority, the point counts of the transmission rate priority andthe total point counts are shown. Further, the point counts of theaddress priority include the point count with respect to the requestsource terminal 10 aa and the point count with respect to thedestination terminal 10 db. The total point count is obtained fromtotaling the higher one of these two point counts of the addresspriority and the point count of the transmission rate priority.

According to the first embodiment, the IP address “1.2.1.2” of the relayapparatus 30 a is “same.same.same.diff” with respect to the IP address“1.2.1.3” of the request source terminal 10 aa, and therefore, in FIG.20, the point count of the address priority is “5”. The IP address“1.2.1.2” of the relay apparatus 30 a is “same.diff.diff.diff” withrespect to the IP address “1.3.2.4” of the destination terminal 10 db,and therefore, in FIG. 20, the point count of the address priority is“1” (i.e., “same.diff.-.-”). The IP address “1.2.2.2” of the relayapparatus 30 b is “same.same.diff.diff” with respect to the IP address“1.2.1.3” of the request source terminal 10 aa, and therefore, in FIG.20, the point count of the address priority is “3” (i.e.,“same.same.diff.-). The IP address “1.2.2.2” of the relay apparatus 30 bis “same.diff.same.diff” with respect to the IP address “1.3.2.4” of thedestination terminal 10 db, and therefore, in FIG. 20, the point countof the address priority is “1” (i.e., “same.diff.-.-”). The IP address“1.3.2.2” of the relay apparatus 30 d is “same.diff.diff.diff” withrespect to the IP address “1.2.1.3” of the request source terminal 10aa, and therefore, in FIG. 20, the point count of the address priorityis “1” (i.e., “same.diff.-.-”). The IP address “1.3.2.2” of the relayapparatus 30 d is “same.same.same.diff” with respect to the IP address“1.3.2.4” of the destination terminal 10 db, and therefore, in FIG. 20,the point count of the address priority is “5”.

Next, returning to FIG. 19, the priority determination part 56 dsearches the priority management table stored in the priority managementDB 5006 (see FIG. 14) based on the maximal transmission rate of each ofthe relay apparatuses 30 managed in the relay apparatus management tablestored in the relay apparatus management DB 5001 (see FIG. 8), anddetermines, for each of the relay apparatuses (30 a, 30 b, 30 d) havingbeen narrowed down in the preliminary narrowing down process, the pointcount of the transmission rate priority (step S46-5). According to thefirst embodiment, since the maximum transmission rate of the relayapparatus 30 a is 100 (Mbps) as shown in FIG. 8, the point count of thetransmission rate priority is “3” when reading the transmission ratepriority shown in FIG. 14. Similarly, since the maximum transmissionrate of the relay apparatus 30 b is 1000 Mbps as shown in FIG. 8, thepoint count of the transmission rate priority is “5”. Similarly, sincethe maximum transmission rate of the relay apparatus 30 d is 10 Mbps asshown in FIG. 8, the point count of the transmission rate priority is“1”.

Next, in step S46-6, the top two relay apparatuses 30 having the highestand second highest point counts from among the total point countsobtained from totaling the higher point counts of the address prioritywith respect to the communication terminals (10 aa, 10 db) and the pointcounts of the transmission rate priority, respectively, are selected bythe preliminary selection part 56 c. According to the example of FIG.20, since the relay apparatuses IDs (111a, 111b, 111d) have the totalpoint counts of “8”, “8”, “6”, respectively, the relay apparatus 30 ahaving the relay apparatus ID “111a” and the relay apparatus 30 b havingthe relay apparatus ID “111b” are thus selected as the top two relayapparatuses 30.

When the narrowing down process in step S46 is thus finished, thetransmission/reception part 51 shown in FIG. 5A transmits relayapparatus narrowing down information, for informing the destinationterminal (10 db) via the communication network 2 of the narrowed downnumber of the relay apparatuses 30, to the destination terminal (10 db)(step S47). The relay apparatus narrowing down information includes thenumber “2” of the relay apparatuses 30 a and 30 b thus narrowed down instep S46, the terminal ID “01aa” of the request source terminal (10 aa)and the above-mentioned selection session ID “se1”. Thereby, thecommunication terminal 10 db can understand the number of the relayapparatuses 30 and the communication terminal from which the request forstarting a TV conference has been carried out, in execution of theselection session of ID “se1”, and also can understand the IP address“1.1.1.2” of the communication management system 50 that is thetransmission source of the relay apparatus narrowing down information.

Then, the transmission/reception part 11 of the communication terminal10 db transmits reception completion information indicating that theabove-mentioned relay apparatus narrowing down information has beenreceived, to the communication management system 50 via thecommunication network 2 (step S48). The reception completion informationincludes the session ID “se1”. Thereby, the communication managementsystem 50 can understand that the transmission of the number of therelay apparatuses 30 carried out in the session ID “se1” has beenfinished, and can know the IP address “1.3.2.4” of the destinationterminal (10 db) that is the transmission source.

Next, with reference to FIG. 21, a process of the destination terminal(10 db) selecting a relay apparatus 30 will be described. First, thecommunication management system 50 transmits advance relay requestinformation that requests relaying in advance, to each of the relayapparatuses 30 a and 30 b narrowed down in step S46 (steps S61-1,S61-2), before a TV conference is started. The advance relay requestinformation includes the session ID “se1”, the IP address “1.2.1.3” ofthe request source terminal (10 aa) and the IP address “1.3.2.4” of thedestination terminal (10 db). Thereby, each of the relay apparatuses 30a and 30 b can understand which selection session is relevant, which isthe request source terminal 10A and which is the destination terminal10B, and also can know the IP address “1.1.1.2” of the communicationmanagement system 50 that is the transmission source of the advancerelay request information.

Next, each of the relay apparatuses 30 a and 30 b transmits, from thetransmission/reception part 31, advance transmission requestinformation, indicating an intention of requesting the request sourceterminal (10 aa) to transmit advance transmission information includinga ping (Packet Internet Groper) described later to the respective one ofthe relay apparatuses 30 a and 30 b itself before a start of a TVconference, to the request source terminal (10 aa) understood in stepsS61-1 and S61-2, via the communication network 2 (steps S62-1, S62-2).The advance transmission request information includes the session ID“se1”. Thereby, the request source terminal (10 aa) can understand it isbeing requested to transmit the advance transmission information to eachof the relay apparatuses 30 a and 30 b in the process of selecting arelay apparatus 30 carried out in the session ID “se1”, and can know theIP addresses “1.2.1.2” and “1.2.2.2” of the relay apparatuses 30 a and30 b that are the transmission sources of the advance transmissionrequest information.

Thus, the IP address of the destination terminal 10 db is not suppliedto the request source terminal 10 aa directly from the communicationmanagement system 50. For example, the IP address of the destinationterminal 10 db is supplied to the relay apparatus 30 a as in step S61-1and the relay apparatus 30 a requests the request source terminal 10 aato transmit the advance transmission information to the own apparatus(the relay apparatus 30 a) as in step S62-1. Not notifying onecommunication terminal 10 of the IP address of another communicationterminal 10 is for ensuring security.

Next, the request source terminal (10 aa) transmits the advancetransmission information to each of the relay apparatuses 30 a and 30 bvia the communication network 2 from the transmission/reception part 11(steps S63-1, S63-2). The advance transmission information isinformation used for measuring a time required from a transmission bythe request source terminal (10 aa) to a reception by the destinationterminal (10 db), as a result of it being transmitted to the destinationterminal (10 db) via each of the relay apparatuses 30 a and 30 b insteadof image data and voice data, in advance to transmission of the actualimage data and voice data. Further, the advance transmission informationincludes ping, used to determine that the request source terminal (10aa), the relay apparatuses 30 a and 30 b and the destination terminal(10 db) are connected in a manner of being able to carry out mutualcommunication, transmission date and time when the advance transmissioninformation has been transmitted from the request source terminal (10aa) and the session ID “se1”. Thereby, each of the relay apparatuses 30a and 30 b can understand that in execution of the session of theselection session ID “se1”, the advance transmission information hasbeen transmitted, and also can understand the IP address “1.2.1.3” ofthe request source terminal (10 aa) that is the transmission source ofthe advance transmission information.

Next, each of the relay apparatuses 30 a, 30 b relays the advancetransmission information to the IP address “1.3.2.4” of the destinationterminal (10 db) included in the advance relay request informationreceived in steps S61-1, S61-2 (steps S64-1, S64-2). Thereby, thedestination terminal (10 db) can understand that in execution of thesession of the session ID “se1”, the advance transmission informationhas been sent, and also can know the IP addresses “1.2.1.2” and“1.2.2.2” of the relay apparatuses 30 a, 30 b that are the transmissionsources (relay sources) of the advance transmission information.

Next, the selection processing part 16 (or finally narrowing down part)of the destination terminal (10 db) finally narrows down to and selectsa single relay apparatus 30 which is to relay image data and voice datain a TV conference, based on the advance transmission information (stepS65).

With reference to FIGS. 5 and 22, the process in step S65 will bedescried in further detail. First, the measurement part 16 a of theselection processing part 16 measures the reception dates and times whenrespective advance transmission information sets, relayed by therespective relay apparatuses 30 a, 30 b, have been received by thetransmission/reception part 11 of the communication terminal 10 db (stepS65-1). Next, the calculation part 16 b calculates the required timesfrom the transmissions to the receptions of the respective advancetransmission information sets, for which the reception dates and timeshave been measured as mentioned above, based on the differences betweenthe reception dates and times and the transmission dates and timesincluded in the respective advance transmission information sets (stepS65-2). Next, the selection part 16 c determines whether in execution ofthe session of the session ID “se1”, all of the advance transmissioninformation sets, corresponding to the number “2” of the relayapparatuses which are the candidates, have been received (step S65-3).When all of the advance transmission information sets have not yet beenreceived (NO), the selection part 16 c determines whether a certain time(here, 1 minute, for example) has elapsed from when the terminal 10 dbhas received the advance transmission information set most recently(step S65-4). In a case where the certain time has not yet elapsed (NO),step S65-1 is returned to. On the other hand, in a case where all of theadvance transmission information sets have been received (step S65-3YES) or the certain time has elapsed (step S65-4 YES), the selectionpart 16 c selects one of the relay apparatuses 30 which has relayed theadvance transmission information set having the shortest required timeamong the required times calculated until now by the calculation part 16b (step S65-5). It is noted that description of the first embodimentwill be carried out for a case where the relay apparatus 30 a isselected as a result of it being determined that the advancetransmission information set relayed by the relay apparatus 30 a has thetime required from the transmission to the reception shorter than thatof the advance transmission information set relayed by the relayapparatus 30 b.

It is noted that in the first embodiment, the destination terminal 10 dbnarrows down the relay apparatuses 30. However, this way is notlimiting, and it is also possible that the destination terminal 10 dbtransmits all the required time information indicating the required timefrom the transmission to the reception of the advance transmissioninformation to the request source terminal 10 aa or the communicationmanagement system 50, which then finally narrows down the relayapparatuses 30 into the single relay apparatus 30 a.

Next, the destination terminal (10 db) transmits selection informationindicating having thus selected the relay apparatus 30 a to thecommunication management system 50 via the communication network 2 fromthe transmission/reception part 11 (step S66). The selection informationincludes the session ID “se1” and the relay apparatus ID “111a” of thethus-selected relay apparatus 30 a. Thereby, the communicationmanagement system 50 can understand that in execution of the session ofthe session ID “se1”, the relay apparatus 30 a has been selected, andalso can know the IP address “1.3.2.4” of the destination terminal (10db) that is a transmission source of the selection information.

Next, the session management part 57 of the communication managementsystem 50 stores the relay apparatus ID “111a” of the finally selectedsingle relay apparatus 30 a in the field area of relay apparatus on therecord including the session ID “se1” in the session management table ofthe session management DB 5005 (see FIG. 12) and manages it (stepS67-1). The transmission/reception part 51 transmits the relay apparatusID “111a” and the IP address “1.3.2.4” of the destination terminal 10 dbto the request source terminal 10 aa (step S67-21). Then, thetransmission/reception part 51 of the communication management system 50transmits relay start request information to indicate an intention ofstarting relaying to the relay apparatus 30 a via the communicationnetwork 2 (step S68). The relay start request information includes therespective IP addresses “1.2.1.3” and “1.3.2.4” of the request sourceterminal 10 aa and the destination terminal 10 db for which relaying isto be carried out. Thus, the relay apparatus 30 a establishes a sessionfor carrying out communication (or remote communication) of three setsof image data of low resolution, medium resolution and high resolution,and voice data (step S69). Thereby, the communication terminals 10 aaand 10 db can start a TV conference.

It is noted that in step S47 mentioned above, the communicationmanagement system 50 transmits the relay apparatus narrowing downinformation to the destination terminal (10 db), and then, thedestination terminal (10 db) carries out the process of selecting therelay apparatus (step S65) through steps S48 through S64-1, S64-2.However, it is not necessary to be limited to this way. Instead, in stepS47, the communication management system 50 may transmit the relayapparatus narrowing down information rather to the request sourceterminal (10 aa), and thus, after that, up to steps S64-1 and S64-2, thetransmission source and destination may be replaced with one anotherbetween the request source terminal (10 aa) and the destination terminal(10 db) for transmitting each information set. Thus, instead of stepS65, the request source terminal (10 aa) may carry out the process ofselecting the relay apparatus. Further, instead of step S66, the requestsource terminal (10 aa) may carry out transmission of the selectioninformation.

Next, with reference to FIGS. 5 and 23, a process of transmitting imagedata and voice data between the request source terminal 10 aa and thedestination terminal 10 db for the purpose of carrying out a TVconference will be described. First, the request source terminal (10 aa)transmits image data of an object photographed by the photographing part14 a and voice data of a voice input by the voice input part 15 a to therelay apparatus 30 a via the communication network 2 from thetransmission/reception part 11 through the image and voice data session“sed” shown in FIG. 2 (step S81). It is noted that according to thefirst embodiment, image data of high image quality including the threeresolutions of the low resolution, the medium resolution and the highresolution, shown in FIGS. 6A, 6B and 6C, and the voice data, aretransmitted. Thus, the relay apparatus 30 a receives the image data ofthe three resolutions and the voice data from the transmission/receptionpart 31. Then, the data quality determination part 33 searches thechange quality management table stored in the change quality managementDB 3001 (see FIG. 7) using the IP address “1.3.2.4” of the destinationterminal (10 db), extracts the corresponding image quality of image datato relay, and thus, recognizes the image quality of image data to relay(step S82). In this case, supposing that the determined image quality ofimage data is “high image quality”, it is the same as the image qualityof the image data received by the transmission/reception part 31.Therefore, the relay apparatus 30 a transfers the image data of theimage quality, as it is, and the voice data of the voice quality, as itis, to the destination terminal (10 db) through the image and voice datasession “sed” (step S83). Thus, the destination terminal (10 db)receives by the transmission/reception part 11 the image data and thevoice data, the display control part 14 b displays the image based onthe received image data on the display device 100, and the voice outputpart 15 b outputs the voice based on the received voice data.

Next, the delay detection part 17 of the terminal 10 db detects a delaytime of reception of the image data received by thetransmission/reception part 11 at certain time intervals (for example,once a second) (step S84). It is noted that description will be carriedout below for a case where the thus-detected delay time is 200 ms. Thedetection of a delay time of reception of the image data may be carriedout by, for example, information indicating transmission date and timewhen the image data is transmitted from the transmission source terminalbeing attached to the image data, and the transmission destinationterminal calculating the delay time based on the difference between thetransmission and reception of the received image data, using theinformation indicating transmission date and time attached to thereceived image data and reception date and time when the received imagedata is received in the transmission destination terminal.

The transmission/reception part 11 of the destination terminal (10 db)transmits delay information indicating the delay time “200 ms” to thecommunication management system 50 via the communication network 2 (stepS85). Thereby, the communication management system 50 can know the delaytime and also know the IP address “1.3.2.4” of the communicationterminal 10 db that is the transmission source of the delay information.

Next, the delay time management part 60 of the communication managementsystem 50 searches the terminal management table stored in the terminalmanagement DB 5003 (see FIG. 10) using the IP address “1.3.2.4” of theabove-mentioned destination terminal (10 db) as a search key, extractsthe corresponding terminal ID “01db”, stores the delay time “200 ms”indicated by the above-mentioned delay information at the field area ofdelay time on the record of the above-mentioned terminal ID “01db” inthe session management table of the session management DB 5005 (see FIG.12), and manages it (step S86)

Next, the quality determination part 58 searches the quality managementtable stored in the quality management DB 5007 (see FIG. 15) using theabove-mentioned delay time “200 ms” as a search key, extracts thecorresponding image quality “medium image quality” of image data, andthus determines the image quality to be “medium image quality” (stepS87).

Next, the transmission/reception part 51 obtains the relay apparatus ID“111a” associated with the above-mentioned terminal ID “01db” from thesession management table stored in the session management DB 5005 (seeFIG. 12), searches the relay apparatus management table stored in therelay apparatus management DB 5001 (see FIG. 8) using the relayapparatus ID “111a” as a search key, and extracts the IP address“1.2.1.2” of the corresponding relay apparatus 30 a (step S88). Then,the transmission/reception part 51 transmits the quality informationindicating the image quality “medium image quality” determined in stepS87 to the relay apparatus 30 a via the communication network 2 (stepS89). The quality information includes the IP address “1.3.2.4” of thedestination terminal (10 db) used as the search key in step S86.Thereby, the change quality management part 34 in the relay apparatus 30a stores the IP address “1.3.2.4” of the communication terminal 10(here, the terminal 10 db) as the transmission destination and the imagequality “medium image quality” of image data to be relayed, in thechange quality management table stored in the change quality managementDB 3001 (see FIG. 7) in a manner of associating them with one another,and manages them (step S90).

Next, the communication terminal 10 aa continuously transmits the imagedata of the high image quality including the three resolutions of thelow resolution, medium resolution and high resolution and voice data tothe relay apparatus 30 a through the image and voice data session “sed”,the same as step S81 mentioned above (step S91). Then, the same as stepS82 mentioned above, the data quality determination part 33 of the relayapparatus 30 a searches the change quality management table stored inthe change quality management DB 3001 (see FIG. 7) using the IP address“1.3.2.4” of the destination terminal (10 db) as a search key, extractsthe corresponding image quality “medium image quality” to relay, andthus recognizes the quality of the image of the image data to relay(step S92). In this example, the determined image quality of image datais “medium image quality” which is lower than the image quality “highimage quality” of the image data received by the transmission/receptionpart 31. Therefore, the data quality changing part 35 reduces the imagequality of image data from “high image quality” to “medium imagequality”, and thus changes the quality of the image of the image data(step S93). Then, the transmission/reception part 31 transmits the imagedata for which the image quality has been thus changed to “medium imagequality” and the voice data for which the voice quality has not beenchanged, to the communication terminal 10 db via the communicationnetwork 2 (step S94). Thus, in the case where the delay in receptionoccurs in the destination terminal (10 db) that receives the image data,the relay apparatus 30 a changes (reduces) the quality of the image toreduce the delay, and thus, it is possible to prevent the personsparticipating in the TV conference from feeling that something is wrong.

Next, using FIG. 29, a process of displaying, at the communicationterminal 10 db, display data indicating a screen page of material dataof the external input apparatus 40 aa connected with the communicationterminal 10 aa and image data input at the communication terminal 10 aawill be described.

Previously, the arrangement information, in which areas and sizes ofimage data and display data to be displayed on the screen of thecommunication terminal 10 are defined, is stored in the communicationterminal 10. Specifically, from the arrangement information providingsystem 80, the transmission/reception part 11 of the communicationterminal 10 db receives the arrangement information via the Internet 2i, the private line 2 cd and the LAN 2 d (step S70-5), and thestoring/reading processing part 19 stores the arrangement information inan area management DB 1002 stored in the storage part 1000 (step S70-6).

When a TV conference has been started, the photographing part 14 a ofthe communication terminal 10 aa takes the picture of the conferenceparticipant and so forth to obtain the image data (step S73). The imagedata obtained in step S73 is transmitted to the relay apparatus 30 a bythe transmission/reception part 11 (step S74-1). The relay apparatus 30a having received the image data converts the quality of the image,(step S75-1), and transmits the image data to the communicationterminals 10 aa and 10 db (steps S76-1 and S76-1′). Then, the respectivetransmission/reception parts 11 of the communication terminals 10 aa and10 db receive the transmitted image data.

Similarly, not shown in FIG. 29, the photographing part 14 a of thecommunication terminal 10 db takes the picture of the conferenceparticipant and so forth to obtain the image data. The image dataobtained is transmitted to the relay apparatus 30 a by thetransmission/reception part 11. The relay apparatus 30 a having receivedthe image data converts the quality of the image, and transmits theimage data to the communication terminals 10 db and 10 aa. Then, therespective transmission/reception parts 11 of the communicationterminals 10 aa and 10 db receive the transmitted image data.

On the other hand, the operation input reception part 42 of the externalinput apparatus 40 aa (see FIG. 27) receives information of at least oneof a material name, a size, an application name and updated date andtime from an operator (step S71-1). Then, the storing/readyingprocessing part 49 searches the material management DB 4001 of thestorage part 4000 using the information thus received by the operationinput reception part 42 as a key, and extracts the correspondingmaterial (step S71-2). Then, the display control part 43 carries outcontrol to display the extracted material on the display device 400(step S71-3), and the display device 400 displays the material data(step S71-4). Next, the display data obtaining part 44 obtains thedisplay data (for displaying the material data that is thus displayed onthe display device 400 in step S71-4) (step S71-5), and thetransmission/reception part 41 transmits the obtained display data tothe external information transmission/reception part 18 of thecommunication terminal 10 aa (step S72).

It is noted that steps S71-1 through S71-4 may be omitted, and in a caseof the omitting, the display data of the desktop screen page istransmitted in step S72. At this time, step S71-5 is carried out inresponse to a transmission starting instruction received from theoperator. The operation input reception part 42 may receive thetransmission starting instruction as a result of a button displayed onthe display device 400 being pressed by the operator. Further, the stepS71-5 may be carried out in response to the external input apparatus 40aa and the communication terminal 10 aa being connected together by acable or such.

The transmission/reception part 11 transmits the display data receivedfrom the external input apparatus 40 aa in step S72 to the relayapparatus 30 a (step S74-2). The relay apparatus 30 a having receivedthe display data converts the quality of the image of the display data(step S75-2), and transmits the display data to the communicationterminals 10 aa and 10 db (steps S76-2, S76-2′). Thus, thetransmission/reception part 11 of the communication terminal 10 dbreceives the display data transmitted from the relay apparatus 30 a, andthe arrangement information selection part 14 c determines thearrangement information (step S79).

Details of the process of converting the quality of the image in stepsS75-1 and S75-2 are the same as those of the process of changing thequality of the image data described above (steps S81 through S94), andduplicate description thereof will be omitted.

Next, using FIG. 30, the process of determining the arrangementinformation carried out in step S79 will be described in detail.

First, the arrangement information selection part 14 c determineswhether the transmission/reception part 11 has received the image data(step S791). When the image data has not been received, the displaycontrol part 14 b displays nothing and the process is finished. When theimage data has been relieved in step S791, the arrangement informationselection part 14 c determines whether the transmission/reception part11 has received the display data (step S792). When the display data hasnot been received in step S792, the storing/reading processing part 19sets a share flag “UNSHARED” (step S793). Then, the arrangementinformation selection part 14 b searches the arrangement informationmanagement table shown in FIG. 24 using the share flag “UNSHARED” as asearch key, and selects the arrangement information “VIEW_MULTI” storedas being associated with the share flag (step S794). Next, the areadetermination part 14 d searches the area management table shown in FIG.25 using the arrangement information “VIEW_MULTI” selected in step S794as a search key and determines the image data sets to be displayed atrespective areas, the information of which is stored as being associatedwith the arrangement information in the area management table, as shownin FIG. 26B where the image data sets of the participants are displayedat the respective areas 1 and 2 (step S795).

Similarly, when the display data has been received in step S792, thestoring/reading processing part 19 determines to select a share flag“SHARED” (step S796). Then, the arrangement information selection part14 b searches the arrangement information management table shown in FIG.24 using the share flag “SHARED” as a search key, and selects thearrangement information “SHARED_MULTI” stored as being associated withthe share flag (step S797). Next, the area determination part 14 dsearches the area management table shown in FIG. 25 using thearrangement information “SHARED_MULTI” selected in step S797 as a searchkey and determines the image data and the display data to be displayedat the respective areas, information of which is stored as beingassociated with the arrangement information in the area managementtable, as shown in FIG. 26A where the material data (i.e., the displaydata) is displayed at the area 1 and the image data sets of theparticipants are displayed at the respective areas 2 and 3 (step S798).

When the arrangement information has been thus determined in step S79,the display control part 14 b displays the image data and the displaydata at the respective areas determined in step S79 (step S80).

Next, a process of stopping transmission of the display data from astate where the external input apparatus 40 aa is transmitting thedisplay data, will be described using FIG. 31.

First, the transmission/reception part 41 of the external inputapparatus 40 aa transmits transmission stopping information for thedisplay data to the communication terminal 10 aa (step S11). Thetransmission stopping information is thus transmitted as a result of theoperation input reception part 11 receiving a transmission stoppinginstruction generated when the user presses a button or such. Further,the transmission stopping information may be thus transmitted as aresult of the cable or such connecting the external input apparatus 40aa and the communication terminal 10 aa together being removed.

When the transmission stopping information is received by thecommunication terminal 10 aa in step S11, the transmission/receptionpart 11 of the communication terminal 10 aa transmits the transmissionstopping information to the relay apparatus 30 in step S12. Thetransmission/reception part 31 of the relay apparatus 30 then transmitsthe transmission stopping information to the communication terminals 10aa and 10 db (step S13, S13′).

When the transmission/reception part 11 of the communication terminal 10aa has received the transmission stopping information (step S13), thestoring/reading processing part 19 sets the share flag “UNSHARED” (stepS14).

Next, the arrangement information selection part 14 c of thecommunication terminal 10 aa uses the share flag “UNSHARED” as a searchkey to search the arrangement information management DB 1001, andextracts the corresponding arrangement information “VIEW_MULTI” (stepS15). Then, the area determination part 14 d uses the arrangementinformation “VIEW_MULTI” extracted in step S15 as a search key to searchthe area management DB 1002 (see FIG. 25), and determines at which areathe image data is to be displayed, the information thereof being storedas being associated with the arrangement information in the areamanagement DB 1002 (step S16). The display control part 14 b displaysthe image data at the respective areas as shown in FIG. 26B where theimage data of the participants are displayed at the respective areas 1and 2 based on the determination (step S17).

Similarly, when the transmission/reception part 11 of the communicationterminal 10 db has received the transmission stopping information (stepS13′), the storing/reading processing part 19 sets the share flag“UNSHARED” (step S14′).

Next, the arrangement information selection part 14 c of thecommunication terminal 10 db uses the share flag “UNSHARED” as a searchkey to search the arrangement information management DB 1001, andextracts the corresponding arrangement information “VIEW_MULTI” (stepS15′). Then, the area determination part 14 d uses the arrangementinformation “VIEW_MULTI” extracted in step S15′ as a search key tosearch the area management DB 1002, and determines at which area theimage data is to be displayed, the information thereof being stored inthe area management DB 1002 as being associated with the arrangementinformation (step S16′). The display control part 14 b displays theimage data at the respective areas shown in FIG. 26B where the imagedata of the participants are displayed at the respective areas 1 and 2based on the determination (step S17′).

Thus, the control of stopping the transmission from the state where theexternal input apparatus 40 aa is transmitting the display data has beenthus described. Control of stopping the transmission from the statewhere the external input apparatus 40 db is transmitting the displaydata is the same as that, and duplicate description will be omitted.

In the first embodiment, based on the share flag which is receptioninformation indicating whether the display data has been received fromthe external input apparatus 40, at which area the image data or thedisplay data is to be displayed is determined. However, instead, atwhich area the image data or the display data is to be displayed may bedetermined according to operations of the user received by the operationinput reception part 12 of the communication terminal 10. Specifically,as shown in FIG. 32, first, it is determined whether the share flag is“SHARED” (step S51). In a case where the share flag is not “SHARED”, theprocess is finished. In a case where the share flag is “SHARED”, thearrangement information to be selected next is determined (step S52)based on the arrangement information corresponding to the currentdisplaying on the display device 100.

Using FIG. 33, details of determining the arrangement information basedon the user's operations will now be described. First, when theoperation input reception part 12 receives the user's operation input ina state where the share flag is “SHARED” (i.e., steps S796→S797 in FIG.30, and thus, the arrangement information is “SHARED_MULTI”) (in FIG.33, (1)), the area management table is searched using the arrangementinformation “VIEW_MULTI” as a search key, and at which area the imagedata is to be displayed is determined, as shown in FIG. 26B where theimage data of the participants are displayed at the respective areas 1and 2. Thus, even when the communication terminal 10 has received thedisplay data from the external input apparatus 40 (i.e., in FIG. 30,step S792 YES→step S796→S797→S798), the display control part 14 b doesnot display the display data, and displays only the image data as shownin FIG. 26B. Further, when from this state, the operation inputreception part 12 receives the user's operation input (FIG. 33, (2)),the area determination table is searched using the arrangementinformation “SHARED_MULTI” as a search key, and at which areas the imagedata and the display data are to be displayed are determined, as shownin FIG. 26A where the display data (material data) is displayed at thearea 1, and the image data of the participants are displayed at therespective areas 2 and 3. Thus, the communication terminal 10 displaysthe display data and the image data as shown in FIG. 26A.

Thus, it is possible to control as to whether the display data is to bedisplayed according to a determination made by the user freely dependingon a situation of the TV conference, even in a case where thecommunication terminal 10 has received the display data.

Main Advantageous Effects of First Embodiment

Thus, according to the first embodiment, when a TV conferenceparticipant considers to use material data (for example, data ofpresentation, word, spreadsheet or such) stored in the external inputapparatus 40 in the TV conference, the participant may connect theexternal input apparatus 40 to the communication terminal 10, thematerial data may be displayed from the screen of the external inputapparatus 40, and the display data thus being displayed from the screenof the external input apparatus 40 may be transmitted to thecommunication terminal 10. Thereby, it is possible to display thedisplay data (material data) on the display device 100 of thecommunication terminal 10 in a large size without carrying out manualoperation.

Further, when the usage of the material data becomes unnecessary, it ispossible to display video data (i.e., the image data) in a large size onthe display device 100 of the communication terminal 10 without carryingout manual operation, by stopping the transmission of the display data.

Therefore, operationality as to whether the material data is necessaryis improved, and also, the TV conference participants can focus theirattention on the data displayed in the large size, and thus, it ispossible for the TV conference to smoothly progress.

Further, in the communication network 2 (see FIG. 1), it may bedifficult to obtain information concerning the environment of theInternet 2 i even if it may be possible to obtain information concerningthe environment of the LAN 2 such as the IP addresses of the relayapparatuses 30 or such. Therefore, according to the first embodiment,the plural relay apparatuses 30 are narrowed down to two or more usingthe information concerning the environment of LAN 2 for relaying theimage data and the voice data. Next, before the image data and the voicedata are actually transmitted between plural communication terminals 10,advance transmission information is transmitted as a trial instead ofthe image data and the voice data. Thereby, it is possible toadvantageously narrow down the relay apparatuses 30 into a single onewhich can relay fastest the advance transmission information.

That is, the top two or more relay apparatuses 30, to which the IPaddresses close to any one of the IP addresses of the communicationterminals 10 are allocated, respectively, are selected. Thereby, it ispossible to select the two or more relay apparatuses 30 as candidatesfor a relay apparatus 30 which is finally used. Then, the advancetransmission information is transmitted between a request sourceterminal 10A and a destination terminal 10B using these candidate relayapparatuses 30. Thereby, from among the two or more relay apparatuses 30as the candidates, it is possible to narrow down to a single one whichhas relayed the advance transmission information with the shortest timerequired for the transmission. Thus, it is possible to advantageouslyachieve transmission of the image data and the voice data at highquality to the utmost limit under the current environment of thecommunication network 2.

Further, according to the first embodiment, when the relay apparatuses30 are narrowed down, not only by preferentially selecting the relayapparatuses 30 having the IP addresses close to the IP addresses of thecommunication terminals 10 that are used for a TV conference, but alsoby considering the maximum transmission rates in the relay apparatuses30, the two more relay apparatuses 30 are selected. Thereby, it ispossible to advantageously narrow down candidates for the relayapparatus 30 suitable to the actual environment of the communicationnetwork 2.

Further, according to the first embodiment, the narrowing down iscarried out from among the relay apparatuses 30 having the operatingstates of on-line. Therefore, it is possible to advantageously narrowdown the candidates for the relay apparatus 30 further suitable to theactual environment of the communication network 2.

Further, according to the first embodiment, it is also possible totransmit the data via the previously determined relay apparatus 30instead of carrying out the process of narrowing down the relayapparatuses 30.

Supplementation of First Embodiment

It is noted that as the conference system, the example of the conferencebetween places distant from each other has been described. However, theconferences include not only one between distant places but also onebetween close places such as those in meeting rooms in one building, onebetween places in one room where a natural voice hardly reachestherebetween, and so forth. That is, the conference means a conferenceto be carried out using conference apparatuses without being carried outdirectly face to face. Further, the conferences include not only a caseof three or more persons but also a case between one person and anotherperson (total two persons).

Each of the communication management system 50 and the program providingsystem 90 according to the first embodiment may be realized by a singlecomputer, or may be realized by plural computers where the respectivefunctional parts (or functions) are divided and are arbitrarilyallocated to the plural computers. Further, in a case where the programproviding system 90 is realized by a single computer, a program to betransmitted by the program providing system 90 may be transmitted ashaving been divided into plural modules, or may be transmitted as nothaving been divided into plural modules. Further, in a case where theprogram providing system 90 is realized by plural computers, dividedplural modules may be transmitted by the plural computers, respectively.

Further, recording media such as CD-ROMs storing the program for thecommunication terminal, the program for the relay apparatus and theprogram for communication management, the HD 204 storing these programs,and the program providing system 90 including the HD 204, are used in acase where the program for the communication terminal, the program forthe relay apparatus and the program for communication management are, asprogram products, provided to users or such in the home country orabroad.

Further, in the above-mentioned first embodiment, the quality of theimage data is managed in particular using the resolution of the image ofthe image data as one example of quality of the image of the image datarelayed by the relay apparatus 30, using the change quality managementtable shown in FIG. 7 and the quality management table shown in FIG. 15.However, the quality of the data is not limited thereto, and as otherexamples of the quality of the data, the data may be managed inparticular using depth (for example, color depth) of the image qualityof the image data, the sampling frequency in the voice data, the bitlength in the voice data, or such. Further, the voice data may betransmitted as having been divided into three resolutions (i.e., highresolution, medium resolution and low resolution).

Further, in FIGS. 8, 10 and 12, the reception dates and times aremanaged. However, it is not necessary to be limited thereto, and it maybe sufficient to manage at least the reception times from among thereception dates and times.

Further, in the above-mentioned embodiments, the IP addresses of therelay apparatuses in FIG. 8 and the IP addresses of the communicationterminals in FIG. 10 are managed. However, it is not necessary to belimited thereto, and as long as relay apparatus identificationinformation for identifying the relay apparatus 30 in the communicationnetwork 2 and terminal identification information for identifying thecommunication terminal 10 in the communication network 2 are used, aFQDN (Fully Qualified Domain Name) thereof may be managed. In this case,the IP address corresponding to the FQDN will be obtained from awell-known DNS (Domain Name System) server. It is noted that “relayapparatus identification information for identifying the relay apparatus30 in the communication network 2” may also be expressed as “relayapparatus connecting destination information indicating a connectingdestination for the relay apparatus 30 in the communication network 2”or “relay apparatus destination information indicating a destination forthe relay apparatus 30 in the communication network 2”. Similarly,“terminal identification information for identifying the communicationterminal 10 in the communication network 2” may also be expressed as“terminal connecting destination information indicating a connectingdestination for the communication terminal 10 in the communicationnetwork 2” or “terminal destination information indicating a destinationfor the communication terminal 10 in the communication network 2”.

Further, in the above-mentioned first embodiment, as an example of thecommunication system 1, the case of the TV conference system has beendescribed. However, it is not necessary to be limited thereto, and anexample of the communication system 1 may be a telephone system for IP(Internet Protocol) telephones, Internet telephones, or such. Further,the transmission system 1 may be an automotive navigation system. Inthis case, one of the communication terminals 10 corresponds to anautomotive navigation apparatus mounted in one automobile, and anotherof the communication terminals 10 corresponds to a management terminalor a management server in a management center which manages theautomotive navigation system, or an automotive navigation apparatusmounted in another automobile. Further, the communication system 1 maybe a communication system for cellular phones. In this case, theterminals correspond to cellular phones.

Further, although the case has been described where the external inputapparatus 40 displays the data from the display device 400, the displaydevice 400 may be an apparatus separate from the external inputapparatus 40, or may be provided to the external input apparatus 40.

Second Embodiment

With reference to FIGS. 34 through 51, a second embodiment of thepresent invention will be described.

Overall Configuration of Second Embodiment

FIG. 34 is a general configuration of a remote communication system 1used for a TV conference according to the second embodiment.

The remote communication system 1 according to the second embodiment isapproximately the same as the communication system 1 according to thefirst embodiment shown in FIG. 1 and described above, and therefore, thesame reference numerals are given to the same or correspondingterminals, apparatuses, systems, routers and so forth, and duplicatedescription will be omitted.

According to the second embodiment, the communication terminals 10 inthe first embodiment are replaced by remote communication terminals 10,respectively, the communication management system 50 in the firstembodiment is replaced by a remote communication management system 50,and the communication network 2 in the first embodiment is replaced by aremote communication network 2. However, the remote communicationterminals 10 according to the second embodiment are approximately thesame as the communication terminals 10 according to the firstembodiment, and the remote communication management system 50 accordingto the second embodiment is approximately the same as the communicationmanagement system 50 according to the first embodiment. The remotecommunication network 2 according to the second embodiment may be thesame as the communication network 2 according to the first embodiment.Therefore, and duplicate description will be omitted.

It is noted that the arrangement information providing system 80 may beomitted from the remote communication system 1 when the remotecommunication terminals 10 previously have the arrangement information.

The external input apparatus 40 transmits the display data, to be usedfor generating a screen page the same as that displayed on the displaydevice 400 of the external input apparatus 40, to the remotecommunication terminal 10. The display data is, for example, image datacreated in a form of JPEG (Joint Photographic Experts Group), Bitmap orsuch, or rendering commands created in a form of GDI (Graphics DeviceInterface). Further, on the entirety or a part of the screen pagecorresponding to the display data, material data which may be used byusing document composition software, spreadsheet software, presentationsoftware (i.e., software used for presentation), or such, may bedisplayed.

Hardware Configuration of Second Embodiment

Next, a hardware configuration of the second embodiment will bedescribed.

FIG. 35 shows a hardware configuration of the remote communicationterminal 10 according to the second embodiment.

The hardware configuration of the remote communication terminal 10according to the second embodiment is approximately the same as thehardware configuration of the communication terminal 10 according to thefirst embodiment described above with reference to FIG. 3, andtherefore, the same reference numerals are given to the same orcorresponding parts, and duplicate description will be omitted.

In the remote communication terminal 10 shown in FIG. 35, a CCD (ChargeCoupled Device) 112 is shown as a specific example of the camera 112shown in FIG. 3. Further, in the ROM 102, a program for the remotecommunication terminal 10 is stored.

The program for the remote communication terminal 10 may be recorded ina form of a file in an installable or executable type in a computerreadable information recording medium such as the above-mentionedrecording medium 106, and may then be circulated.

The CCD 112 is a solid state image sensing device which converts lightinto electric charges, and obtaining an electronic signal from an imageof an object. As long as a device is capable of photographing an object,it is not necessary to be limited to the CCD, and a CMOS (ComplementaryMetal Oxide Semiconductor) device or such may be used instead.

FIG. 36 shows a hardware configuration of the remote communicationmanagement system 50 according to the second embodiment.

The hardware configuration of the remote communication management system50 according to the second embodiment may be the same as the hardwareconfiguration of the communication management system 50 according to thefirst embodiment described above with reference to FIG. 4, and duplicatedescriptions will be omitted. It is noted that in the ROM 202, a programfor remote communication management (i.e., for the remote communicationmanagement system 50) is stored.

It is noted that the program for remote communication management may berecorded in a computer readable recording medium such as theabove-mentioned recording medium 206, CD-ROM 213 or such in a form of afile of an installable type or an executable type, and then becirculated.

Further, the relay apparatus 30 has the same hardware configuration asthat of the remote communication management apparatus 50, and therefore,the description thereof will be omitted. In the ROM 202 of the relayapparatus 30, a program for the relay apparatus for controlling therelay apparatus 30 is recorded. Also in this case, the program for therelay apparatus may be recorded in a computer readable recording mediumsuch as the above-mentioned recording medium 206, CD-ROM 213 or such ina form of a file of an installable type or an executable type, and becirculated.

The external input apparatus 40 has the same hardware configuration asthat of the remote communication management apparatus 50, and therefore,the description thereof will be omitted. However, in the ROM 202 of theexternal input apparatus 40, a program for material management forcontrolling the external input apparatus 40 is recorded. Also in thiscase, the program for material management may be recorded in a computerreadable recording medium such as the recording medium 206 in a form ofa file of an installable type or an executable type, and be circulated.

Further, each of the arrangement information providing system 80 and theprogram providing system 90 has the same hardware configuration as thatof the communication management apparatus 50, and therefore, thedescription thereof will be omitted. However, in the ROM 202 of thearrangement information providing system 80, a program for providingarrangement information is recorded. Similarly, in the ROM 202 of theprogram providing system 90, a program for providing programs isrecorded. Also in this case, each of these programs for providingarrangement information and providing programs may be recorded incomputer readable recording medium such as the recording medium 206 orthe CD-ROM 213 in a form of a file of an installable type or anexecutable type, and be circulated.

It is noted that each of the above-mentioned programs may be recorded ina computer readable recording medium such a CD-R (Compact DiscRecordable), a DVD (Digital Versatile Disk) or a Blu-ray Disc as anotherexample of the above-mentioned detachable recording medium, and then beprovided.

Functional Configuration of Second Embodiment

Next, a functional configuration of the second embodiment will bedescribed. FIGS. 37A and 37B show a functional block diagram of each ofthe remote communication terminals 10, each of the relay apparatuses 30and the remote communication management system 50 included in the remotecommunication system 1 according to the second embodiment. In FIGS. 37Aand 37B, the remote communication terminal 10, the relay apparatus 30and the remote communication management system 50 are connected togetherby the remote communication network 2 in such a manner that datacommunication can be mutually carried out. The external input apparatus40 is connected with the remote communication terminal 10 in such amanner that data communication can be mutually carried out. FIGS. 6A, 6Band 6C, described above for the first embodiment, also illustrate imagequality according to the second embodiment. FIG. 7, described above forthe first embodiment, also shows a concept of an example of a changequality management table according to the second embodiment. FIG. 8,described above for the first embodiment, also shows a concept of anexample of a relay apparatus management table according to the secondembodiment. FIG. 9, described above for the first embodiment, also showsa concept of an example of a terminal authentication management tableaccording to the second embodiment. FIG. 10, described above for thefirst embodiment, also shows a concept of an example of a terminalmanagement table according to the second embodiment. FIG. 11, describedabove for the first embodiment, also shows a concept of an example of adestination list management table according to the second embodiment.FIG. 12, described above for the first embodiment, also shows a conceptof an example of a session management table according to the secondembodiment. FIG. 13, described above for the first embodiment, alsoshows a concept of an example of an address priority management tableaccording to the second embodiment. FIG. 14, described above for thefirst embodiment, also shows a concept of an example of a transmissionrate priority management table according to the second embodiment. FIG.15, described above for the first embodiment, also shows a concept of anexample of a quality management table according to the secondembodiment. FIG. 47 shows a functional block diagram of the externalinput apparatus 40 included in the remote communication system 1according to the second embodiment. FIG. 28, described above for thefirst embodiment, also shows a concept of a material management tableaccording to the second embodiment.

<Functional Configuration of Remote Communication Terminal>

As shown in FIG. 37B, the remote communication terminal 10 includes atransmission/reception part 11, an operation input reception part 12, alogin request part 13, a photographing part 14 a, a display control part14 b, an arrangement information extraction part (i.e., arrangementinformation selection part) 14 c, a voice input part 15 a, a voiceoutput part 15 b, a finally narrowing down part (or selection processingpart) 16, a delay detection part 17, an external informationtransmission/reception part 18, and a storing/reading processing part19. The respective parts correspond to functions or functioning partsrealized as a result of the respective elements/components shown in FIG.35 operating according to instructions given by the CPU 101 thatoperates according to the program for the communication terminal storedin the ROM 102. Further, the communication terminal 10 has a storagepart 1000 built by any one of the ROM 102, RAM 103 and the SSD 105 shownin FIG. 35.

(Arrangement Information Management Table)

In the storage part 1000, an arrangement information management DB(DataBase) 1001 including an arrangement information management tablesuch as that shown in FIG. 45 is stored. In the arrangement informationmanagement table, in association with reception information indicatingwhether the current state is one where the remote communication terminal10 can receive display data or whether the remote communication terminal10 has received display data, arrangement information that indicatesdata (information) to be displayed at each area on a screen is stored.Whether the current state is one where the remote communication terminal10 can receive display data is determined by, for example, whether theremote communication terminal 10 is connected with the external inputapparatus 40 by a USB cable or such. In a case where the remotecommunication terminal 10 is connected with the external input apparatus40, the current state is one where the remote communication terminal 10can receive display data, and thus, the reception information mayindicate “received”. In a case where the remote communication terminal10 is not connected with the external input apparatus 40, the currentstate is one where the remote communication terminal 10 cannot receivedisplay data, and thus, the reception information may indicate “notreceived”. As shown in the example of FIG. 45, the arrangementinformation management table shows that in a case where the receptioninformation indicates “received” indicating that display data has beenreceived, the received display data is displayed at an “area 1”; imagedata obtained of photographing by the photographing part 14 a of theremote communication terminal 10 aa which is a terminal at the other endof the communication is displayed at an “area 2”; and image dataobtained from photographing by the photographing part 14 a of thecommunication terminal 10 db which is the own terminal is displayed atan “area 3”. Further, the arrangement information management table showsthat in a case where the reception information indicates “not received”indicating that display data has not been received, image data obtainedof photographing by the photographing part 14 a of the communicationterminal 10 aa which is a terminal at the other end of the communicationis displayed at the “area 1”; image data obtained from photographing bythe photographing part 14 a of the communication terminal 10 db which isthe own terminal is displayed at the “area 2”; and nothing is displayedat the “area 3”. The areas 1, 2 and 3 are determined by sizes andpositions at the screen, respectively. Examples shown in FIGS. 46A and46B show examples where based on arrangement drawings that are thearrangement information stored in the arrangement information managementDB 1001, the material data (display data) and/or the image data aredisplayed. FIG. 46A shows a screen page displayed based on thearrangement drawing 1 in the case where the reception informationindicates “received”. FIG. 46B shows a screen page displayed based onthe arrangement drawing 2 in the case where the reception informationindicates “not received”. As shown in FIGS. 46A and 46B, the area at theleft side is referred to as the area 1, the area at the top right isreferred to as the area 2, and the area at the middle right is referredto as the area 3.

The display data means information corresponding to the screen pagewhere material data is being displayed on the display device 400 of theexternal input apparatus 40 connected to the remote communicationterminal 10. It is noted that the material data (or display data) meansdata created using document composition software, spreadsheet softwareor such, and is distinguished from image data obtained fromphotographing by the camera CCD shown in FIG. 35.

(Detailed Functional Configuration of Remote Communication Terminal)

Next, the respective parts of the remote communication terminal 10 willbe described.

The transmission/reception part 11, the operation input reception part12 and the login request part 13 may be the same as those according tothe first embodiment described above, and duplicate description will beomitted.

The photographing part 14 a is realized by the CCD 112 and the imagingdevice I/F 113 shown in FIG. 35, photographs an object, and outputsimage data thus obtained from the photographing.

The display control part 14 b may be the same as that according to thefirst embodiment described above, and duplicate description will beomitted.

The arrangement information extraction part 14 c extracts thearrangement information indicating an arrangement in a screen page to bedisplayed on the display device 100 by the display control part 11,based on the reception information indicating whether thetransmission/reception part 11 has received the display data.

The voice input part 15 a and voice output part 15 b may be the same asthose according to the first embodiment described above, and duplicatedescription will be omitted.

The finally narrowing down part (or selection processing part) 16 may bethe same as that according to the first embodiment described above, andduplicate description will be omitted.

The delay detection part 17, the external informationtransmission/reception part 18 and the storing/reading processing part19 may be the same as those according to the first embodiment describedabove, and duplicate description will be omitted.

It is noted that the terminal ID and a relay apparatus ID describedlater according to the second embodiment indicate identificationinformation such as a language, a character/letter, a sign or varioussorts of marks used for uniquely identifying the remote communicationterminal 10 and the relay apparatus 30, respectively. Further, each ofthe terminal ID and the relay apparatus ID may be identificationinformation that is a combination of at least two of the above-mentionedlanguage, character/letter, sign and various sorts of marks. Further,hereinafter, the remote communication terminal 10 as a request sourcewhich requests a start of a TV conference will be referred to as a“request source terminal 10A” and the remote communication terminal 10as a destination as a request destination will be referred to as a“destination terminal 10B”.

<Functional Configuration of Relay Apparatus>

Functions and respective parts of the relay apparatus 30 will now bedescribed. As shown in FIG. 37B, the relay apparatus 30 includes atransmission/reception part 31, a state detection part 32, a dataquality determination part 33, a change quality management part 34, adata quality changing part 35 and a storing/reading management part 39.These parts correspond to functions or functioning parts realized as aresult of the respective elements/components shown in FIG. 36 operatingaccording to instructions given by the CPU 201 that operates accordingto the program stored in the ROM 202. Further, the relay apparatus 30has a storage part 3000 (storing a change quality management DB 3001)comprising the HDD 205 shown in FIG. 36.

The relay apparatus 30 shown in FIG. 37B may be the same as thataccording to the first embodiment described above with reference to FIG.5B, and duplicate description will be omitted.

<Functional Configuration of External Input Apparatus>

The external input apparatus 40 includes, as shown in FIG. 47, atransmission/reception part 41, an operation input reception part 42, adisplay control part 43, a display data obtaining part 44, a connectiondetection part 45, an installation determination part 46, a mountingpart 47, a program obtaining part 48 and a storing/reading processingpart 49. These parts are functions or functional parts shown in FIG. 36realized as a result of the CPU 201 operating according to the program(i.e., the program for material management) stored in the ROM 202.Further, the external input apparatus 40 has a storage part 4000comprising the HDD 205 shown in FIG. 36.

(Material Management Table)

In the storage part 4000, a material management DB (DataBase) 4001including a material management table such as that shown in FIG. 28,described above for the first embodiment, is stored. The materialmanagement DB (DataBase) 4001 may be the same as that of the firstembodiment described above with reference to FIG. 28, and duplicatedescription will be omitted.

<Detailed Functional Configuration of External Input Apparatus>

Next, each part of the external input apparatus 40 will be described indetail.

The transmission/reception part 41, operation input reception part 42,display control part 43 and display data obtaining part 44 may be thesame as those of the first embodiment described above with reference toFIG. 27, and duplicate description will be omitted.

The connection detection part 45 detects that the external inputapparatus 40 has been connected with another apparatus by a USB cable orsuch. The installation determination part 46 determines whether thedisplay data obtaining part 44 has been installed in the external inputapparatus 40. The mounting part 47 mounts a storage part of the otherapparatus, connected with the external input apparatus 40 by the USBcable or such, to the external input apparatus 40. The program obtainingpart 48 obtains the display data obtaining part 44 from the thus-mountedstorage part of the other apparatus and installs the obtained displaydata obtaining part 44 in the external input apparatus 40.

The storing/reading processing part 49 may be the same as that of thefirst embodiment described above with reference to FIG. 27, andduplicate description will be omitted.

<Functional Configuration of Remote Communication Management System>

Next, functions or functional parts of the remote communicationmanagement system 50 will be described. As shown in FIG. 37A, the remotecommunication management system 50 includes a transmission/receptionpart 51, a terminal authentication part 52, a state management part 53,a terminal extraction part 54, a terminal state obtaining part 55, anarrowing down part 56, a session management part 57, a qualitydetermination part 58, a storing/reading processing part 59 and a delaytime management part 60. These parts correspond to functions orfunctioning parts realized as a result of the respectiveelements/components shown in FIG. 36 operating according to instructionsgiven by the CPU 201 that operates according to the program for remotecommunication management stored in the ROM 202. Further, the remotecommunication management system 50 has a storage part 5000 (storing arelay apparatus management DB 5001, a terminal authentication managementDB 5002, a terminal management DB 5003, a destination list management DB5004, a session management DB 5005, a priority management DB 5006 and aquality management DB 5007) comprising the HD 204 shown in FIG. 36.

The remote communication management system 50 shown in FIG. 37A may bethe same as that according to the first embodiment described above withreference to FIG. 5A, and duplicate description will be omitted.

Processing/Operations of Second Embodiment

Thus, the configuration and functions or functional parts of the remotecommunication system according to the second embodiment has beendescribed. Next, with reference to FIGS. 38 through 43, and FIGS. 48through 51, a processing method in the remote communication system 1according to the second embodiment will be described.

FIG. 38 is a sequence diagram showing a process of managing theinformation indicating the state of each of the relay apparatuses 30.The process of managing the information indicating the states of therelay apparatuses 30, transmitted from the corresponding relayapparatuses 30 to the remote communication management system 50,respectively, shown in FIG. 38, may be the same as that of the firstembodiment described above with reference to FIG. 16, and the same stepnumbers are given to the same steps, and duplicate description will beomitted.

FIG. 39 is a sequence diagram showing a process of a preparation stagefor starting remote communication between plural remote communicationterminals 10. The process of a preparation stage for starting remotecommunication between the remote communication terminals 10 aa and 10 dbshown in FIG. 39 may be the same as that of the first embodimentdescribed above with reference to FIG. 17, and the same step numbers aregiven to the same steps, and duplicate description will be omitted.

FIG. 40 is a sequence diagram showing a process of narrowing down therelay apparatuses 30. FIG. 41 is a flowchart showing the process ofnarrowing down the relay apparatuses 30. FIG. 20, also described abovefor the first embodiment, shows a state of calculation of point countsused when narrowing down the relay apparatuses 30. The process ofnarrowing down the relay apparatuses 30 shown in FIG. 40 may be the sameas that of the first embodiment described above with reference to FIG.18, and the same step numbers are given to the same steps, and duplicatedescription will be omitted. The process of step S46 shown in FIG. 41may be the same as that of the first embodiment described above withreference to FIG. 19 and FIG. 20, and therefore, the same step numbersare given to the same steps, and duplicate description therefor will beomitted.

FIG. 42 is a sequence diagram showing the process of selecting a relayapparatus 30 by the remote communication terminal 10. FIG. 43 is aflowchart showing the process of selecting a relay apparatus 30 by theremote communication terminal 10. The process of selecting a relayapparatus 30 by the remote communication terminal 10 aa (destinationterminal) may be the same as that of the first embodiment describedabove with reference to FIG. 21, and the same step numbers are given tothe same steps, and duplicate description will be omitted. The processof step S65 shown in FIG. 43 may be the same as that of the firstembodiment described above with reference to FIG. 22, and the same stepnumbers are given to the same steps, and duplicate description will beomitted.

FIG. 44 is a sequence diagram showing a process of transmitting imagedata and voice data between the remote communication terminals 10. Theprocess of transmitting image data and voice data between the remotecommunication terminals 10 aa and 10 db for the purpose of carrying outa TV conference may be the same as that of the first embodimentdescribed above with reference to FIG. 23, and the same step numbers aregiven to the same steps, and duplicate description will be omitted.

FIG. 45 shows a concept of the arrangement information management tablestored by the storage part 1000 according to the second embodiment.FIGS. 46A and 46B show the examples of screen pages in which thematerial data and the image data are displayed based on the arrangementdrawings as the arrangement information stored in the storage part 1000according to the second embodiment. FIG. 48 is a sequence diagramshowing a process of displaying the image data and the display data bythe remote communication terminal 10. FIG. 49 is a flowchart showing aprocess of determining, by the remote communication terminal 10, thearrangement information used for displaying the image data and thedisplay data. FIGS. 50A, 50B show examples of images displayed on theremote communication terminal 10. FIG. 51 shows an example of an imagedisplayed on the remote communication terminal 10.

Next, using FIG. 48, a process of displaying, at the communicationterminal 10 db, the display data for indicating a screen page ofmaterial data of the external input apparatus 40 aa connected with thecommunication terminal 10 aa and the image data input from thecommunication terminal 10 aa will be described.

When the external input apparatus 40 aa and the remote communicationterminal 10 aa are connected together, the connection detection part 45of the external input apparatus 40 aa detects it (step S70-1). When theconnection detection part 45 of the external input apparatus 40 aa thusdetects that the external input apparatus 40 aa and the remotecommunication terminal 10 aa have been connected together, theinstallation determination part 46 determines whether the display dataobtaining part 44 has been already installed (step S70-2). When it isdetermined that the display data obtaining part 44 has not beeninstalled yet in step S70-2, the mounting part 47 of the external inputapparatus 40 aa mounts the storage part 1000 of the remote communicationterminal 10 aa (step S70-2), the program obtaining part 48 obtains thedisplay data obtaining part 44 stored in the storage part 1000, andinstalls it (step S70-4).

In a case where it is determined in step S70-2 that the display dataobtaining part 44 has been already installed, the process starting fromstep S71-1 is proceeded to.

Previously, the arrangement information in which areas and sizes ofdisplaying the image data and the display data on the screen of theremote communication terminal 10 are defined is stored in the remotecommunication terminal 10. Specifically, from the arrangementinformation providing system 80, the transmission/reception part 11 ofthe remote communication terminal 10 db receives the arrangementinformation via the Internet 2 i, the private line 2 cd and the LAN 2 d(step S70-5), and the storing/reading processing part 19 stores thearrangement information in the arrangement information management DB1001 stored in the storage part 1000 (step S70-6).

When a TV conference has been started, the photographing part 14 a ofthe remote communication terminal 10 aa takes a picture of theconference participant or such to obtain the image data (step S73). Theimage data obtained in step S73 is transmitted to the relay apparatus 30a by the transmission/reception part 11 (step S74-1). The relayapparatus 30 a having received the image data converts the quality ofthe image (step S75-1), and transmits the image data to the remotecommunication terminal 10 db (step S76-1). Then, thetransmission/reception part 11 of the remote communication terminal 10db receives the image data transmitted from the relay apparatus 30 a,and extracts the arrangement information used for displaying, from thearrangement information management DB 1001 (step S77).

Further, the relay apparatus 30 a transmits the image data of the imageconverted in step S75-1 also to the remote communication terminal 10 aa(step S76-1′). The transmission/reception part 11 of the remotecommunication terminal 10 aa receives the image data transmitted fromthe relay apparatus 30 a, and carries out the process the same as thatcarried out by the remote communication terminal 10 db in steps S77 andS80.

On the other hand, the operation input reception part 42 of the externalinput apparatus 40 aa (see FIG. 27) receives information of at least oneof a material name, a size, an application name and updated date andtime from the operator (step S71-1). Then, the storing/readyingprocessing part 49 searches the material management DB 4001 of thestorage part 4000 using the information received by the operation inputreception part 42 as a search key, and extracts the correspondingmaterial (step S71-2). Then, the display control part 43 requests todisplay the extracted material on the display device 400 (step S71-3),and the display device 400 displays the material data (step S71-4).Next, the display data obtaining part 44 obtains the display data of ascreen page including the display of the material data thus requested bythe display control part 43 to display on the display device 400 in stepS71-3 (step S71-5), and the transmission/reception part 41 transmits theobtained display data to the external information transmission/receptionpart 18 of the communication terminal 10 aa (step S72). Thetransmission/reception part 11 transmits the display data that has beenthus received in step S72 to the relay apparatus 30 a (step S74-2). Therelay apparatus 30 a having received the display data converts thequality of the image of the display data (step S75-2), and transmits thedisplay data (the quality of the image of which has been thus converted)to the communication terminal 10 db (step S76-2). Then, thetransmission/reception part 11 of the communication terminal 10 dbreceives the display data transmitted from the relay apparatus 30 a, andextracts the arrangement information for displaying, from thearrangement information management DB 1001 (step S77).

Further, the relay apparatus 30 a transmits the display data of theimage converted in step S75-2 also to the remote communication terminal10 aa (step S76-2′). The transmission/reception part 11 of the remotecommunication terminal 10 aa receives the image data transmitted fromthe relay apparatus 30 a, and carries out the process the same as thatcarried out by the remote communication terminal 10 db in steps S77 andS80.

Details of the process of converting the quality of the image in stepsS75-1 and S75-2 are the same as those of the process of changing thequality of the image data described above (steps S81 through S94), andduplicate description thereof will be omitted.

Next, using FIG. 49, the process of extracting the arrangementinformation carried out in step S77 will be described in detail.

First, the arrangement information extraction part 14 c determineswhether the transmission/reception part 11 has received the image data(step S791). When the image data has not been received, the displaycontrol part 14 b displays nothing and the process is finished. When theimage data has been relieved in step S791, the arrangement informationextraction part 14 c determines whether the transmission/reception part11 has received the display data (step S792). When the display data hasreceived in step S792, the arrangement information management tableshown in FIG. 45 is searched using the reception information “received”as a search key, and the arrangement information (corresponding to“arrangement drawing 1” such as that shown in FIG. 46A) indicating thatthe display data and the image data sets are associated with therespective areas (area 1, area 2 and area 3) is extracted (step S796).When the display data has not received in step S792, the arrangementinformation management table shown in FIG. 45 is searched usingreception information “not received” as a search key, and thearrangement information (corresponding to “arrangement drawing 2” suchas that shown in FIG. 46B) indicating that the image data sets areassociated with the respective areas (area 1 and area 2) is extracted(step S793).

Then, the display control part 14 b creates images based on the imagedata and the display data and displays the created images on thecorresponding respective areas on the screen according to thearrangement information extracted in step S793 or S796 (step S80).

Next, to change the areas at which image data and display data aredisplayed will now be described.

When the reception information is “received”, in step S80, as shown inFIG. 46A, the image based on the display data is displayed at the area1; the image based on the image data transmitted from the remotecommunication terminal 10 aa of the other participant of the TVconference is displayed at the area 2; and the image photographed by theremote communication terminal 10 db of the user is displayed at the area3. Then, when the user inputs an operation to the remote communicationterminal 10 db to change the images to be displayed at the respectiveareas of the screen page, then, as shown in FIG. 50A, the image based onthe image data transmitted from the remote communication terminal 10 aaof the other participant of the TV conference is displayed at the area1; the image photographed by the remote communication terminal 10 db ofthe user is displayed at the area 2; and the image based on the displaydata is displayed at the area 3. Then, when the user again inputs anoperation to the remote communication terminal 10 db to change theimages to be displayed at the respective areas of the screen page, asshown in FIG. 50B, the image photographed by the remote communicationterminal 10 db of the user is displayed at the area 1; the image basedon the display data is displayed at the area 2; and the image based onthe image data transmitted from the remote communication terminal 10 aaof the other participant of the TV conference is displayed at the area3. Then, when the user further again inputs an operation to the remotecommunication terminal 10 db to change the images to be displayed at therespective areas of the screen, the respective images are displayed atthe areas in the same arrangement as that where the respective imagehave been displayed in step S80. It is noted that the input of theoperation is received by the operation input reception part 12 that isrealized by, for example, an arrow key or such.

When the reception information is “not received”, in step S80, as shownin FIG. 46B, the image based on the image data transmitted from theremote communication terminal 10 aa of the other participant of the TVconference is displayed at the area 1; and the image photographed by theremote communication terminal 10 db of the user is displayed at the area2. After that, in a case where voice data has been obtained from themicrophone 114 of the remote communication terminal 10 db of the user,as shown in FIG. 51, the image photographed by the remote communicationterminal 10 db of the user is displayed at the area 1; and the imagebased on the image data transmitted from the remote communicationterminal 10 aa of the other participant of the TV conference isdisplayed at the area 2. In a case where voice data that has beenobtained from the microphone 114 of the remote communication terminal 10aa of the other participant of the TV conference has been then receivedby the transmission/reception part 11 of the remote communicationterminal 10 db of the user, as shown in FIG. 46B, the image based on theimage data transmitted from the remote communication terminal 10 aa ofthe other participant of the TV conference is displayed at the area 1;and the image photographed by the remote communication terminal 10 db ofthe user is displayed at the area 2.

Main Advantageous Effects of Second Embodiment

Thus, according to the second embodiment, based on whether the remotecommunication terminal has received the display data that is displayedon the external input apparatus of another participant of a TVconference, displaying is controlled using the arrangement informationextracted from the storage part. Thereby, it is possible to effectivelyuse the screen of the display device.

Further, when the external input apparatus 40 is connected, the imageindicated by the display data is displayed larger than the imageindicated by the image data. Thereby, the participant of the TVconference can easily pay attention to material or such prepared for theTV conference.

Further, when the external input apparatus 40 is connected, the imageindicated by the display data is displayed at the center more than theimage indicated by the image data. Thereby, the participant of the TVconference can easily pay attention to material or such prepared for theTV conference.

Further, according to the second embodiment, in a case where the imagesof persons or such indicated by the image data are to be given priorityin comparison to the image indicated by the display data, the screenpage can be switched in response to the user's instruction, and thereby,it is possible to display the images of the persons or such indicated bythe image data lager than the image indicated by the display data.

Supplementation of Second Embodiment

It is noted that as the conference system, the example of the conferencebetween places distant from each other has been described. However, theconferences include not only one between distant places but also onebetween close places such as those in meeting rooms in one building, onebetween places in one room where a natural voice hardly reachestherebetween, and so forth. That is, the conference means a conferenceto be carried out using conference apparatuses without being directlycarried out, face to face. Further, the conferences include not only acase of three or more persons but also a case between one person andanother person (total two persons).

Further, it is possible to control or limit the number of items of thematerial data. Thereby, it is possible to save resources. Further, inthis case, it is possible to display that the number of items of thematerial data is controlled or limited. Thereby, it is possible toimprove convenience of the user.

Further, even in a case where no material data is displayed on thescreen of the display device, a process of sharing the display data iscarried out. For example, in a case where only a desktop screen page isdisplayed on the external input apparatus, this screen page itselfcorresponds to the display data.

Further, whether the remote communication terminal is in a state wherethe display can be received may be determined from whether theapplication used for realizing the display data obtaining part 44 orsuch has been started up. In a case where the application has beenstarted up, the remote communication terminal is in a state of beingable to receive the display data, and the reception data “received” maybe set. In a case where the application has not been started up, theremote communication terminal is in a state of not being able to receivethe display data, and the reception data “not received” may be set.Further, it is also possible to set a state of being able to receive thedisplay data after the application used for obtaining the display dataobtaining part 44 or such has been started up. In this case, by theapplication, on the display device 400 of the external input apparatus40, a menu of “start sharing”, “stop sharing” or such is displayed, theuser is urged to select one of these menu items, and thereby, thereception information of “received” or “not received” may be determined.That is, when the user has selected “start sharing”, the receptioninformation “received” may be set. When the user has selected “stopsharing”, the reception information “not received” may be set.

Each of the communication management system 50 and the program providingsystem 90 according to the second embodiment may be realized by a singlecomputer, or may be realized by plural computers where the respectiveparts (or functions) are divided and are arbitrarily allocated to theplural computers. Further, in a case where the program providing system90 is realized by a single computer, a program to be transmitted by theprogram providing system 90 may be transmitted as having been dividedinto plural modules, or may be transmitted as not having been dividedinto plural modules. Further, in a case where the program providingsystem 90 is realized by plural computers, divided plural modules may betransmitted by the plural computers, respectively.

Further, recording media such as CD-ROMs storing the program for theremote communication terminal, the program for the relay apparatus andthe program for remote communication management, the HD 204 storingthese programs, and the program providing system 90 including the HD204, are used in a case where the program for the remote communicationterminal, the program for the relay apparatus and the program for remotecommunication management, as program products, are provided to users orsuch in the home country or aboard.

Further, in the above-mentioned second embodiment, quality of image datais managed in particular using resolution of an image of image data asone example of quality of an image of image data relayed by the relayapparatus 30, using the change quality management table shown in FIG. 7and the quality management table shown in FIG. 15. However, quality ofdata is not limited thereto, and as other examples of quality of data,data may be managed in particular using depth (for example, color depth)of image quality of image data, a sampling frequency in voice data, abit length in voice data, or such. Further, voice data may betransmitted as having been divided into three resolutions (i.e., highresolution, medium resolution and low resolution).

Further, in FIGS. 8, 10 and 12, the reception dates and times aremanaged. However, it is not necessary to be limited thereto, and it maybe sufficient that at least the reception times from among the receptiondates and times are managed.

Further, in the above-mentioned embodiments, the IP addresses of therelay apparatuses in FIG. 8 and the IP addresses of the communicationterminals in FIG. 10 are managed. However, it is not necessary to belimited thereto, and as long as relay apparatus identificationinformation for identifying the relay apparatus 30 in the remotecommunication network 2 or terminal identification information foridentifying the remote communication terminal 10 in the remotecommunication network 2 is used, a FQDN (Fully Qualified Domain Name)thereof may be managed. In this case, the IP address corresponding tothe FQDN will be obtained from a well-known DNS (Domain Name System)server. It is noted that “relay apparatus identification information foridentifying the relay apparatus 30 in the remote communication network2” may also be expressed as “relay apparatus connecting destinationinformation indicating a connecting destination for the relay apparatus30 in the remote communication network 2” or “relay apparatusdestination information indicating a destination for the relay apparatus30 in the remote communication network 2”. Similarly, “terminalidentification information for identifying the remote communicationterminal 10 in the remote communication network 2” may also be expressedas “terminal connecting destination information indicating a connectingdestination for the remote communication terminal 10 in the remotecommunication network 2” or “terminal destination information indicatinga destination for the remote communication terminal 10 in the remotecommunication network 2”.

Further, in the above-mentioned embodiments, as an example of thecommunication system 1, the case of the TV conference system has beendescribed. However, it is not necessary to be limited thereto, and anexample of the remote communication system 1 may be a telephone systemfor IP (Internet Protocol) telephone, Internet telephone, or such.Further, the transmission system 1 may be an automotive navigationsystem. In this case, one of the remote communication terminals 10corresponds to an automotive navigation apparatus mounted in oneautomobile, and another of the remote communication terminals 10corresponds to a management terminal or a management server in amanagement center which manages the automotive navigation system, or anautomotive navigation apparatus mounted in another automobile. Further,the remote communication system 1 may be a communication system forcellular phones. In this case, the terminals correspond to cellularphones.

Further, although the case has been described where the external inputapparatus 40 displays data from the display device 400, the displaydevice 400 may be an apparatus separate from the external inputapparatus 40, or may be provided to the external input apparatus 40.

Although this invention has been described in detail with reference tocertain embodiments, variations and modifications exist within the scopeand spirit of the invention as described and defined in the followingclaims.

1. A communication terminal which receives display data for displayingcertain data on a screen of a display device and image data of anoptical-to-electrical-converted image, from another communicationterminal via a communication network, comprising: a storage partconfigured to store reception information indicating whether the displaydata has been received by the communication terminal and arrangementinformation indicating at least one of a size and a position of theimage based on the image data, in a manner of associating with oneanother; a reception part configured to receive the display data and theimage data; an extraction part configured to extract, based on thereception information, the corresponding arrangement information fromthe storage part; and a display control part configured to display theimage using the image data based on the arrangement informationextracted by the extraction part.
 2. The communication terminal asclaimed in claim 1, wherein when the reception information indicatesthat the reception part has received the display data, the displaycontrol part is configured to display the image corresponding to thereceived display data according to an arrangement based on thearrangement information larger than the image corresponding to thereceived image data according to an arrangement based on the arrangementinformation.
 3. The communication terminal as claimed in claim 2,wherein when the reception information indicates that the reception parthas received the display data, the display control part is configured todisplay the image corresponding to the received image data based onfirst arrangement information that is certain arrangement information ofthe arrangement information stored in the storage part, when thereception information indicates that the reception part has not receivedthe display data, the display control part is configured to display theimage corresponding to the received image data based on secondarrangement information of the arrangement information stored in thestorage part, different from the first arrangement information, and thedisplay control part is configured to display the image corresponding tothe image data displayed in the arrangement based on the secondarrangement information larger than the image corresponding to the imagedata displayed in the arrangement based on the first arrangementinformation.
 4. The communication terminal as claimed in claim 3,wherein the display control part is configured to display the imagecorresponding to the image data displayed in the arrangement accordingto the second arrangement information at a position closer to the centerthan the image corresponding to the image data displayed in thearrangement according to the first arrangement information.
 5. Thecommunication terminal as claimed in claim 1, further comprising: aninstruction reception part configured to receive an instructionconcerning a position or a size to display the image indicated by theimage data and the image indicated by the display data, wherein thedisplay control part is configured to change the position or the size todisplay the images indicated by the image data and the display databased on the instruction received by the instruction reception part. 6.The communication terminal as claimed in claim 5, wherein theinstruction reception part is configured to receive an instruction toreplace positions of the images indicated by the image data and theimage indicated by the display data with one another, and when theinstruction reception part has received the instruction, the displaycontrol part is configured to replace the positions of the imageindicated by a first set of image data and the image indicated by asecond set of image data of plural sets of the image data and the imageindicated by the display data with each other.
 7. A communication methodcarried out by a communication terminal which receives display data fordisplaying certain data on a screen of a display device and image dataof an optical-to-electrical-converted image, from another communicationterminal via a communication network, wherein the communication terminalcomprises a storage part configured to store reception informationindicating whether the display data has been received by thecommunication terminal and arrangement information indicating at leastone of a size and a position of the image based on the image data, in amanner of associating with one another, and the communication methodcomprises: receiving the display data and the image data; extracting,based on the reception information, the corresponding arrangementinformation from the storage part; and displaying the image using theimage data based on the arrangement information extracted by theextracting.
 8. A computer readable information recording medium storinga communication program executed by a communication terminal whichreceives display data for displaying certain data on a screen of adisplay device and image data of an optical-to-electrical-convertedimage, from another communication terminal via a communication network,wherein the communication terminal comprises a storage part configuredto store reception information indicating whether the display data hasbeen received by the communication terminal and arrangement informationindicating at least one of a size and a position of the image based onthe image data, in a manner of associating with one another, and thecommunication program, when executed by one or plural processors,performs: receiving the display data and the image data; extracting,based on the reception information, the corresponding arrangementinformation from the storage part; and displaying the image using theimage data based on the arrangement information extracted by theextracting.